New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GnuPG 2.1 & dirmngr handling #15

Open
morxa opened this Issue Jul 2, 2015 · 22 comments

Comments

Projects
None yet
9 participants
@morxa
Copy link
Contributor

morxa commented Jul 2, 2015

Starting with gnupg 2.1.4 on Arch Linux, receiving a key always fails with the same error:

May 17 18:55:36 clavius parcimonie.sh[435]: > Sleeping 11282 seconds before refreshing key <key_id>...
May 17 22:03:38 clavius parcimonie.sh[435]: gpg: keyserver receive failed: IPC syntax error

Updating the key manually works as expected:

$ gpg --recv-keys "<key_id>"
gpg: key <short_id>: "..." not changed
gpg: Total number processed: 1
gpg:              unchanged: 1

With gnupg 2.1.3, this error did not occur.

My gpg.conf:

require-cross-certification
keyserver hkp://pool.sks-keyservers.net
personal-digest-preferences SHA512
cert-digest-algo SHA512
default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed
use-agent

My /etc/parcimonie.sh.d/user.conf:

PARCIMONIE_USER=<user>
USE_RANDOM=true

My tor config is the default of an up-to-date Arch Linux. Since tor was not updated with gnupg, I don't think the error is directly related to tor.

How to reproduce

I can only guess that it must be related to gnupg 2.1.4, as my configuration is probably the standard configuration:

  1. install gnupg >= 2.1.4
  2. start parcimonie
  3. check the log for an IPC syntax error

Expected behavior

The key should be refreshed successfully.

Actual behavior

Receiving the key fails with the error shown above.

@EtiennePerot

This comment has been minimized.

Copy link
Owner

EtiennePerot commented Jul 2, 2015

Hmm, odd. Can you try running gnupg manually with torsocks and seeing what happens? Something like (adjust as needed):

torsocksConfig="/tmp/tor.$RANDOM"
echo "TorAddress 127.0.0.1" > "$torsocksConfig"
echo "TorPort 9050" >> "$torsocksConfig"
TORSOCKS_CONF_FILE="$torsocksConfig" torsocks gnupg --recv-keys "$keyID"

That should more-or-less reproduce what parcimonie.sh is trying to run. Maybe it fails while running under torsocks only. If so, make sure your system is fully up-to-date (no partial update) and file a torsocks bug.

@morxa

This comment has been minimized.

Copy link
Contributor Author

morxa commented Jul 11, 2015

It works when manually running gpg through torsocks:

% ./parcimonie-test.sh 0xA611742C                                                                                       :(
gpg: key A611742C: "Till Hofmann <hofmanntill@gmail.com>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1

Where parciomie-test.sh looks as follows:

#!/bin/sh
torsocksConfig="/tmp/tor.$RANDOM"
echo "TorAddress 127.0.0.1" > "$torsocksConfig"
echo "TorPort 9050" >> "$torsocksConfig"
TORSOCKS_CONF_FILE="$torsocksConfig" torsocks gpg2 --recv-keys "$1"

Furthermore, I can reproduce the error on a fresh Fedora 22 install. If I configure parcimonie to use gpg2, I get the same error message (key ID replaced):

Jul 11 15:37:41 clavius.hal9000 parcimonie.sh[21102]: > Sleeping 15 seconds before refreshing key ABDCEF0123456789...
Jul 11 15:37:56 clavius.hal9000 parcimonie.sh[21102]: gpg: keyserver receive failed: IPC syntax error

Interestingly, when using gnupg 1.4.19 (the default gpg on Fedora 22), it works, but it gives some warnings I do not understand:

Jul 11 15:39:46 clavius.hal9000 parcimonie.sh[22433]: > Sleeping 12 seconds before refreshing key EF45510680FB02326B045AFB32474CF834EC9CBA...
Jul 11 15:39:58 clavius.hal9000 parcimonie.sh[22433]: gpg: requesting key 34EC9CBA from hkp server pool.sks-keyservers.net
Jul 11 15:40:01 clavius.hal9000 parcimonie.sh[22433]: gpg: DBG: armor-keys-failed (KEY 0xEF45510680FB02326B045AFB32474CF834EC9CBA BEGIN
Jul 11 15:40:01 clavius.hal9000 parcimonie.sh[22433]: ) ->0
Jul 11 15:40:01 clavius.hal9000 parcimonie.sh[22433]: gpg: DBG: armor-keys-failed (KEY 0xEF45510680FB02326B045AFB32474CF834EC9CBA END
Jul 11 15:40:01 clavius.hal9000 parcimonie.sh[22433]: ) ->0
Jul 11 15:40:01 clavius.hal9000 parcimonie.sh[22433]: gpg: key 34EC9CBA: "Fedora (23) <fedora-23-primary@fedoraproject.org>" 2 new signatures
Jul 11 15:40:01 clavius.hal9000 parcimonie.sh[22433]: gpg: public key EC6F945C is 273 seconds newer than the signature
Jul 11 15:40:01 clavius.hal9000 parcimonie.sh[22433]: gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
Jul 11 15:40:01 clavius.hal9000 parcimonie.sh[22433]: gpg: depth: 0  valid:   3  signed:  12  trust: 0-, 0q, 0n, 0m, 0f, 3u
Jul 11 15:40:01 clavius.hal9000 parcimonie.sh[22433]: gpg: public key EC6F945C is 273 seconds newer than the signature
Jul 11 15:40:01 clavius.hal9000 parcimonie.sh[22433]: gpg: depth: 1  valid:  12  signed:  26  trust: 5-, 0q, 0n, 6m, 1f, 0u
Jul 11 15:40:01 clavius.hal9000 parcimonie.sh[22433]: gpg: next trustdb check due at 2015-09-30
Jul 11 15:40:01 clavius.hal9000 parcimonie.sh[22433]: gpg: Total number processed: 1
Jul 11 15:40:01 clavius.hal9000 parcimonie.sh[22433]: gpg:         new signatures: 2
@Mirclus

This comment has been minimized.

Copy link

Mirclus commented Jul 11, 2015

I investigated a little: The command called by the script is TORSOCKS_CONF_FILE=/... torsocks gpg --batch --with-colons --keyserver-options http-proxy= --recv-keys 0123456789ABC....

If i run this with --debug-all I get this output:

gpg: reading options from '/home/me/.gnupg/gpg.conf'
gpg: enabled debug flags: packet mpi crypto filter iobuf memory cache memstat trust hashing cardio ipc clock lookup extprog
gpg: DBG: [not enabled in the source] start
gpg: DBG: chan_3 <- # Home: /home/me/.gnupg
gpg: DBG: chan_3 <- # Config: /home/me/.gnupg/dirmngr.conf
gpg: DBG: chan_3 <- OK Dirmngr 2.1.6 at your service
gpg: DBG: connection to the dirmngr established
gpg: DBG: chan_3 -> OPTION http-proxy=
gpg: DBG: chan_3 <- ERR 167772436 IPC syntax error <Dirmngr> - option argument expected
gpg: DBG: chan_3 -> BYE
gpg: keyserver receive failed: IPC syntax error
gpg: DBG: [not enabled in the source] stop
gpg: random usage: poolsize=600 mixed=0 polls=0/0 added=0/0
              outmix=0 getlvl1=0/0 getlvl2=0/0
gpg: secmem usage: 0/32768 bytes in 0 blocks

If I drop the http-proxy-option, the update succeeds.

Additional information:

local/parcimonie-sh-git 16.9523a3e-1
core/gnupg 2.1.6-1
community/torsocks 2.1.0-1

~ % cat ~/.gnupg/dirmngr.conf 
hkp-cacert /home/me/.gnupg/sks-keyservers.netCA.pem
@EtiennePerot

This comment has been minimized.

Copy link
Owner

EtiennePerot commented Jul 12, 2015

The default http-proxy= setting was introduced as part of solving #10. I guess this is a GnuPG 1 vs GnuPG 2 issue. Not really sure how to fix it without doing GnuPG version detection, or attempting to read the user's GnuPG config file and replicate it without the http-proxy directive that it may have. Neither solution seems very clean.

It would also be possible to make users have to specify all keyserver-options in GNUPG_KEYSERVER_OPTIONS and then ignoring those defined in the GnuPG config file (slightly cleaner to do than just trying to remove the http-proxy part of it), but that may silently break some users' setup if the use a custom cert or whatnot. I'd personally prefer that though because it's a bunch of trouble to keep this working while respecting external configuration files' keyserver-options.

Opinions?

@MeisterP

This comment has been minimized.

Copy link
Contributor

MeisterP commented Jul 12, 2015

From "[Announce] GnuPG 2.1.0 "modern" released" (https://lists.gnupg.org/pipermail/gnupg-announce/2014q4/000358.html)

  • The Dirmngr is now part of GnuPG proper and also takes care of
    accessing keyserver.

Running

torsocksConfig="/tmp/tor.$RANDOM"
echo "TorAddress 127.0.0.1" > "$torsocksConfig"
echo "TorPort 9050" >> "$torsocksConfig"
TORSOCKS_CONF_FILE="$torsocksConfig" torsocks gnupg --recv-keys "$keyID"

works fine without an error. If I run

gnupg --recv-keys "$keyID

right after that, I still get tor traffic. I guess that's because the dirmngr daemon is still up and running with torsocks...

Also, dirmngr only partially supports the http-proxy options see "dirmngr ignores honor-http-proxy and http-proxy options" (https://bugs.gnupg.org/gnupg/issue1786)

@morxa

This comment has been minimized.

Copy link
Contributor Author

morxa commented Jul 12, 2015

Changing http-proxy= to http-proxy=none works with GnuPG 2, but that doesn't work with GnuPG 1.4. So we would still need to do GnuPG version detection, but the only difference between the two versions would be the argument passed to http-proxy.

@EtiennePerot

This comment has been minimized.

Copy link
Owner

EtiennePerot commented Jul 12, 2015

In order to keep ensuring that all key refreshes are done on unique Tor circuits, dirmngr needs to run for exactly one key refresh, and then be killed right afterwards. We also need to make sure that gpg doesn't use a potentially-already-running-but-without-torsocks instance of dirmngr that the user may have launched manually in the past...

I think this means having to do the following, if a dirmngr binary is installed:

  • Make a unique, temporary GNUPGHOME directory somewhere in /tmp, let's call it /tmp/parcimonie_gpg (with $RANDOM appended to it or something), chmod'd 700.
  • Symlink all files from the real GNUPGHOME to /tmp/parcimonie_gpg, except S.dirmngr.
  • Run GNUPGHOME=/tmp/parcimonie_gpg TORSOCKS_CONF_FILE=/tmp/tor.something torsocks dirmngr --daemon --homedir /tmp/parcimonie_gpg (for some reason dirmngr ignores --homedir when considering where to create the socket file).
  • Run GNUPGHOME=/tmp/parcimonie_gpg DIRMNGR_INFO=/tmp/parcimonie_gpg/S.dirmngr TORSOCKS_CONF_FILE=/tmp/tor.something torsocks gpg --recv-keys the_one_key_id. Thanks to the symlinks, gpg is actually editing the real keyring files.
  • kill whatever the PID of the dirmngr invocation was.
  • rm -r /tmp/parcimonie_gpg

That's a pretty roundabout way to go about it but I can't really think of a way that preserves parcimonie.sh's properties without preventing normal gpg usage, and taking into consideration the bugs that dirmngr currently has.

@genodeftest

This comment has been minimized.

Copy link
Contributor

genodeftest commented Jan 20, 2016

One could probably use namespaces instead of creating a temporary directory and symlink most files into it. See e.g. unshare(1) or nsenter(1). Downside: Linux-specific feature (I don't know how much you care about *BSD or *nix).

@genodeftest

This comment has been minimized.

Copy link
Contributor

genodeftest commented Feb 17, 2016

In gpg2, gpg.conf provides a dirmngr-program option. It can be used to torify dirmngr 1

Wouldn't it be sufficient (although not perfect) to create a torifying wrapper for dirmngr that terminates dirmngr after some (e.g. 5) seconds?

@loudfishmonger

This comment has been minimized.

Copy link

loudfishmonger commented Aug 10, 2016

I'm a new user. I just experienced this same issue.

Running this, I get the IPC syntax error:

TARGET_REFRESH_TIME=10 MIN_WAIT_TIME=1 parcimonie.sh

If I set http-proxy=none, as suggested above, it works fine.

TARGET_REFRESH_TIME=10 MIN_WAIT_TIME=1 GNUPG_KEYSERVER_OPTIONS="http-proxy=none" parcimonie.sh

My question is, when I set GNUPG_KEYSERVER_OPTIONS="http-proxy=none" like this, is it overwriting any keyserver-options I may have in my gpg.conf? I don't set http-proxy in that file, but I do have a couple other things. Is the best current solution to this issue to just set the proxy like I am in my second example, or am I better off filling GNUPG_KEYSERVER_OPTIONS with whatever keyserver-options I have set in my config?

pigmonkey added a commit to pigmonkey/spark that referenced this issue Aug 12, 2016

set no-honor-keyseserver-url for parcimonie
We need to set GNUPG_KEYSERVER_OPTIONS and pass something to address
EtiennePerot/parcimonie.sh#15

I set no-honor-keyserver-url in my gpg.conf anyway. I think the option
makes sense, so we'll use that until the parcimonie issue is resolved.
@Feandil

This comment has been minimized.

Copy link

Feandil commented Nov 22, 2016

Maybe a temporary work-around (until a more complete solution can be written) could be to kill all running dirmngr just before and after a key is refreshed? The probability that this would take place as the same time as the user's gpg activity would be rather low.

@ilf

This comment has been minimized.

Copy link

ilf commented Dec 10, 2016

Sorry for not providing a fix, but still running into this 17 months after it was first reported, I would love to boost morale and thank everyone working on fixing this. :) Thanks!

@EtiennePerot

This comment has been minimized.

Copy link
Owner

EtiennePerot commented Dec 10, 2016

@loudfishmonger This set up is not safe, because it reuses Tor circuits across key refreshes. By setting http-proxy=none, dirmngr goes through Tor (by virtue of having been launched through the torsocks'd gpg binary), and then keeps running. During future key refreshes, dirmngr is not restarted, so the currently-running one will refresh keys potentially over the same circuit that it used to refresh the first one.

  • @Feandil's workaround sounds pretty easy but still has the potential for race conditions.
  • @genodeftest's suggestion of using namespaces might help but it is indeed Linux-only. With no easy way to know how the non-Linux usage is, it's not really viable either.
  • @genodeftest's suggestion of creating a self-destructing dirmngr wrapper is good, but incomplete, as we still need to make sure that gpg actually uses it rather than a potentially already-running one that was launched without wrapper.
  • My own suggestion of creating a symlinked mirror of ~/.gnupg is also flawed in that it could lead to some weird file write race conditions if both parcimonie.sh's gpg and the user's own gpg try to write to the home directory at the same time.

tl;dr: There are no perfect answers.

One gleam of hope is that dirmngr actually has a --use-tor option. It's not safe to use either (per the man page, it leaks DNS) but it shows that the gnupg folks take this sort of use case in consideration.

I propose to implement the following compromise. It is an extension of @Feandil's suggestion but with more implementation details, and protection against race conditions that could result in non-Tor usage.

Preferences:

  • Add a DIRMNGR_PATH option to specify the path to dirmngr.
  • Add a DIRMNGR_CLIENT_PATH option to specify the path to dirmngr-client.

On start:

  • Check if DIRMNGR_PATH is set.
    • If it is unset, try to find it in $PATH.
      • If not found, assume GnuPG < 2.1.0 and disable all of the functionality described in this comment.
      • If found, set DIRMNGR_PATH to the full path to dirmngr.
    • If it is set, check that it exists and is executable. If not, exit with an error code.
  • Check if DIRMNGR_CLIENT_PATH is set.
    • If it is unset, try to find it in $PATH.
      • If not found, but DIRMNGR_PATH is valid, exit with an error code.
      • If found, set DIRMNGR_CLIENT_PATH to the full path to dirmngr-client.
    • If it is set, check that it exists and is executable. If not, exit with an error code.

On key refresh:

  • Check if dirmngr is running using dirmngr-client --ping.
    • If it is, run the following in a loop:
      • killall dirmngr
      • sleep 1
      • dirmngr-client --ping
      • If the dirmngr-client --ping command fails, break out of the loop.
      • If we've looped for over a minute, give up this key refresh.
  • Run a Torified dirmngr in our target GNUPGHOME. Take note of its PID.
  • Loop until dirmngr-client --ping succeeds to make sure we have started it correctly.
    • If we've looped for over a minute, give up this key refresh.
  • Verify that there exists only a single dirmngr process running as the current user, and that its PID matches the one of the dirmngr that we launched earlier. This ensures that the dirmngr we launched with the correct Torification options is running and is the one responding on queries to the socket inside our target GNUPGHOME, free of race conditions. (There's still some rare condition that could happen due to PID reuse, but that would be extremely rare, especially considering the likelihood of the reused PID being used by a new dirmngr of all things.)
    • If the verification fails, give up this key refresh.
  • Refresh the key.
  • Run killall dirmngr.
  • Wait until the PID of the dirmngr we launched disappears.
  • Done!

This is still subject to a race condition in the form of the user manually refreshing a key while parcimonie.sh itself is in the process of refreshing one. However, this is much better than the risk of a race condition in which we could reuse a non-Torified dirmngr).

If there are no objections, I plan to implement this sometime this month.

@EtiennePerot EtiennePerot self-assigned this Dec 10, 2016

@ilf

This comment has been minimized.

Copy link

ilf commented Dec 11, 2016

One gleam of hope is that dirmngr actually has a --use-tor option. It's not safe to use either (per the man page, it leaks DNS) but it shows that the GnuPG folks take this sort of use case in consideration.

DNS is only leaked if DNS is used. For --use-tor, we should use the Tor Hidden Service hkp://jirk5u4osbsr34t5.onion from https://sks-keyservers.net/overview-of-pools.php, which solves this problem.

@EtiennePerot

This comment has been minimized.

Copy link
Owner

EtiennePerot commented Dec 11, 2016

Doing that would force all users to use the same keyserver rather than being able to specify their own, which is a regression. It also doesn't solve the circuit reuse problem.

@ilf

This comment has been minimized.

Copy link

ilf commented Dec 12, 2016

Doing that would force all users to use the same keyserver

Not quite. sks-keyservers.net sais:

An experimental Tor OnionBalance hidden service is running as hkp://jirk5u4osbsr34t5.onion consisting of the servers marked with Tor support in the status list as backend.

Currently, that Tor pool consists of 15 of the 94 servers in the SKS pool, tendency: growing.
The GnuPG default keyserver is keys.gnupg.net, which is a CNAME for pool.sks-keyservers.net.
I think the SKS keyserver onion qualifies as a good default.

Also, with onion, circuit reuse becomes less of a problem, since the threat level is an attacker on the server-side re-identifying a client re-using the same exit note. But an onion-server never sees an exit node, it only sees "a Tor user". Also, not only the client, but also the server could use a new curcuit after the random sleep. This is a good enough compromise for me not to need a guarantee for a new circuit on each connection.

@ilf

This comment has been minimized.

Copy link

ilf commented Jan 3, 2017

One gleam of hope is that dirmngr actually has a --use-tor option. It's not safe to use either (per the man page, it leaks DNS) but it shows that the GnuPG folks take this sort of use case in consideration.

From GnuPG 2.1.17, dirmngr now uses Libdns instead of ADNS, which doesn't leak DNS any more: http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=commitdiff;h=5a4a109354d53cf3673d0636731c67021d3f367a

@EtiennePerot

This comment has been minimized.

Copy link
Owner

EtiennePerot commented Jan 24, 2017

Thanks @ilf, that's good to know. Unfortunately it means either doing version detection again, or just hoping that users will never use the prior versions... Using --use-tor or torsocks doesn't really make much of a difference to the proposed solution, and given that torsocks works for all GnuPG versions without DNS leaks, I don't see much of a reason to favor --use-tor over it.

As for the .onion keyserver pool, that's ultimately orthogonal to this issue. It's a good default though. Maybe parcimonie.sh could detect if the user has no default keyserver set up, and use that one in this case. That's a separate feature request from this issue though.

@EtiennePerot EtiennePerot changed the title IPC syntax error with gnupg 2.1.4 GnuPG 2.1 & dirmngr handling Jan 24, 2017

EtiennePerot added a commit that referenced this issue Jan 28, 2017

Add configuration options for dirmngr, with autodetection as fallback.
This currently does nothing, but it will be used to switch between the two modes of operation described in issue #15, depending on whether dirmngr should be used (GnuPG ≥ 2.1) or not (GnuPG < 2.1).
@genodeftest

This comment has been minimized.

Copy link
Contributor

genodeftest commented Jan 19, 2018

Even with the changes in 5aa21ef, I'm getting the IPC syntax error. Any idea what could be done here?

Software versions (Fedora 27):
gnupg2-2.2.3-1.fc27.x86_64
gnupg-1.4.22-3.fc27.x86_64
tor-0.3.1.9-1.fc27.x86_64

My parcimonie.sh.service file in ~/.config/systemd/user:

[Unit]
Description=parcimonie with config file /etc/parcimonie.sh.d/%i.conf

[Service]
Type=simple
#EnvironmentFile=/etc/parcimonie.sh.d/%i.conf
ExecStart=/usr/bin/parcimonie.sh

[Install]
WantedBy=default.target

In ~/.gnupg/dirmngr.conf, use-tor is NOT enabled.
After terminating dirmngr, running gpg2 --recv-key 0xsomeid works fine.

@EtiennePerot

This comment has been minimized.

Copy link
Owner

EtiennePerot commented Jan 20, 2018

@genodeftest That commit doesn't actually change behavior; it only implements the logic for finding dirmngr and dirmngr-client. The variables it creates (dirmngrPath, dirmngrClientPath) are not used by the rest of the script. I've been dragging my feet on this.

CNG added a commit to CNG/dotfiles that referenced this issue Apr 3, 2018

set no-honor-keyseserver-url for parcimonie
We need to set GNUPG_KEYSERVER_OPTIONS and pass something to address
EtiennePerot/parcimonie.sh#15

I set no-honor-keyserver-url in my gpg.conf anyway. I think the option
makes sense, so we'll use that until the parcimonie issue is resolved.
@morxa

This comment has been minimized.

Copy link
Contributor Author

morxa commented Nov 27, 2018

Fedora is updating the default GnuPG to GnuPG2.

I'm wondering whether we should also default parcimonie to GnuPG2, or if we should set GnuPG1 as default for the Fedora package? Currently, the standard configuration for parcimonie in Fedora uses GnuPG1.

@gnoutchd

This comment has been minimized.

Copy link

gnoutchd commented Jan 22, 2019

(perl) parcimone has also had a miserable time with dirmngr. Antoine Beaupré suggested ditching dirmngr and talking to the keyservers ourselves, observing that HKP is not hard. It's really just a bit of HTTP.

So, something like this?

recvKey() {
  local fingerprint="$1"

  local tmpdir="$(mktemp -d)"
  local gpgtmp="${tmpdir}/gnupghome"
  mkdir "${gpgtmp}"
  chmod 0700 "${gpgtmp}"

  local ok=no
  curl -x "socks5h://parcimonie-${RANDOM}:parcimone-${RANDOM}@127.0.0.1:9050/" \
    "http://jirk5u4osbsr34t5.onion/pks/lookup?op=get&search=0x${fingerprint}" \
    >"${tmpdir}/response.dat" && \
  gpg --homedir "${gpgtmp}" --import "${tmpdir}/response.dat" && \
  gpg --homedir "${gpgtmp}" --output "${tmpdir}/response-filtered.dat" \
    --export "0x${fingerprint}" && \
  gpg --import "${tmpdir}/response-filtered.dat" && \
  ok=yes

  rm -r "${tmpdir}"
  [[ "$ok" == "yes" ]]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment