invoke GPG unattended with passphrase

[gonewest818]

The purpose of this pull request is to ask if the approach will be accepted before I finish up the integration and testing.

The concept is to offer "gpg --passphrase-fd 0" as an alternate way to supply a passphrase, for example to perform unattended jar signing in TravisCI or similar.

So the key changes were to add some syntax for specifying the passphrase in the project.clj via an environment variable (which can be encrypted in a way that won't be revealed in Travis CI). And then some enhancement to the way gpg is launched to supply that passphrase on stdin. A few unit tests were added too.

If folks are okay with this approach I will probably close the pull request and keep working to complete the implementation, and then reopen a new pull request with the complete change when I think it's closer to done.

thanks

[gonewest818]

By the way, the builds are failing but I don't think this PR has anything to do with it.

Travis job #2454.1 is failing with errors like this:

Could not transfer artifact clojure-complete:clojure-complete:jar:0.2.4 from/to clojars (https://repo.clojars.org/): Received fatal alert: protocol_version

which appears to be a TLS issue, and based on your recent commit history it looks like you're aware of it. See also this and this.

Travis job #2454.2 is failing on the same dependency a few different times:

lein test :only leiningen.test.uberjar/test-uberjar-managed-dependencies
ERROR in (test-uberjar-managed-dependencies) (core.clj:4617)
Uncaught exception, not in assertion.
expected: nil
  actual: clojure.lang.ExceptionInfo: Uberjar aborting because jar failed: Problem opening jar /home/travis/.m2/repository/rome/rome/0.9/rome-0.9.jar

but I just deleted my ~/.m2/repository/rome and ran the same test by hand to prove to myself lein would retrieve the missing jar.

[technomancy]

Go ahead and ignore Travis; it has been failing for no reason. However, I think I fixed the problem with the test suite on master, and if you rebase off the latest it should help. I hope to take a look at the rest of the PR in the next couple days.

[technomancy]

As long as the documentation is clear that putting the GPG passphrase in the environment is meant only for unattended deploys, I don't see a problem with this in principle. However, @hypirion has spent more time working on stdin issues, so his input would be more useful on specific implementation details.

[gonewest818]

Ok I'll wait for further feedback.

[gonewest818]

note - when I test this locally the signature generation is working as expected, but then I'm getting errors during the attempt to send signed artifacts into clojars (no checksum provided?). I didn't touch any checksum logic and actually I don't even see where it's done. Is that happening inside aether?

Could not transfer metadata org.clojars.gonewest818:defcon/maven-metadata.xml from/to clojars (https://clojars.org/repo): Access denied to: https://clojars.org/repo/org/clojars/gonewest818/defcon/maven-metadata.xml, ReasonPhrase: Forbidden - no checksums provided for defcon-0.6.1.pom.asc.

Failed to deploy metadata: Could not transfer metadata org.clojars.gonewest818:defcon/maven-metadata.xml from/to clojars (https://clojars.org/repo): Access denied to: https://clojars.org/repo/org/clojars/gonewest818/defcon/maven-metadata.xml, ReasonPhrase: Forbidden - no checksums provided for defcon-0.6.1.pom.asc.

(edit) Furthermore, the error occurs only with release versions, whereas with snapshots the unsigned pom, jar, and maven-metadata.xml are uploaded with md5 and sha1 checksums as usual. @cemerick, any hints where to look?

[gonewest818]

Further note - deploying signed artifacts seems broken on the master branch too. I retried the same signed deploy as above using a fresh checkout on commit afd3dee and got exactly the same error as reported above:

Could not transfer metadata org.clojars.gonewest818:defcon/maven-metadata.xml from/to clojars (https://clojars.org/repo): Access denied to: https://clojars.org/repo/org/clojars/gonewest818/defcon/maven-metadata.xml, ReasonPhrase: Forbidden - no checksums provided for defcon-0.6.2.jar.asc.
Failed to deploy metadata: Could not transfer metadata org.clojars.gonewest818:defcon/maven-metadata.xml from/to clojars (https://clojars.org/repo): Access denied to: https://clojars.org/repo/org/clojars/gonewest818/defcon/maven-metadata.xml, ReasonPhrase: Forbidden - no checksums provided for defcon-0.6.2.jar.asc.

I backed up to commit 781195f0 (the one before the aether update on May 26) and tested again, and that deploy worked as expected. I'm wondering if the new aether is really working ok?

I'm going to stop testing because this "unattended signatures" feature seems to work (for me) but I'd feel much better about my testing if the deploys fully worked.

[technomancy]

Tracking the signature checksum problem separately at #2303; thanks for pointing that out.

[gonewest818]

Thanks for resolving #2303. Because of that, my local testing is now successful.

I rebased and squashed the commits to clean up the git history. During the rebase I had to manually merge with #2293 because @glts changed some of the same code, but I left it in a way that should preserve his work as well as mine. I also added documentation per your earlier request.

If you and @hypirion (and @glts) are satisfied, then I think this is ready to merge?

[gonewest818]

Well. I'm finding Travis and CircleCI default to relatively old distributions of Ubuntu with either gpg 1.4 or gpg 2.0, and this implementation uses the --pinentry-mode option introduced in gpg 2.1. So things are fine on my laptop, not so much in CI. I'm going to have to do a little more to support those older gpg versions (or at least detect them and fail gracefully).

[gonewest818]

With those latest commits the code now detects which version of gpg is available and provides the right gpg signing arguments in each case.

[gonewest818]

Hey... so now that 2.8.x is released what are your thoughts about this PR?

[technomancy]

Sorry for the delay on this. If we can't get any other input on this I'll go ahead and bring it in so we can at least get more people using it.

Thanks!

[gonewest818]

Cool. I just helped a little with unattended deploys in the Travis CI configs for cider-nrepl and orchard. I don’t know if @bbatsov plans to gpg sign the release jars, but if he does then this is timely.

[technomancy]

This broke deploys in 2.8.2, so I had to revert it.

$ lein deploy clojars
Created /home/phil/src/leiningen/lein-pprint/target/lein-pprint-1.2.1-SNAPSHOT.jar
Wrote /home/phil/src/leiningen/lein-pprint/pom.xml
java.lang.ClassCastException: clojure.lang.Keyword cannot be cast to java.lang.String
 at cemerick.pomegranate.aether$authentication.invokeStatic (aether.clj:173)
    cemerick.pomegranate.aether$authentication.invoke (aether.clj:173)
    cemerick.pomegranate.aether$set_authentication.invokeStatic (aether.clj:184)
    cemerick.pomegranate.aether$set_authentication.invoke (aether.clj:181)
    cemerick.pomegranate.aether$make_repository.invokeStatic (aether.clj:207)
    cemerick.pomegranate.aether$make_repository.invoke (aether.clj:200)
    cemerick.pomegranate.aether$deploy_artifacts$fn__316.invoke (aether.clj:361)
    clojure.core$map$fn__4785.invoke (core.clj:2644)
    clojure.lang.LazySeq.sval (LazySeq.java:40)
    clojure.lang.LazySeq.seq (LazySeq.java:49)
    clojure.lang.LazySeq.first (LazySeq.java:71)
    clojure.lang.RT.first (RT.java:667)
    clojure.core$first__4339.invokeStatic (core.clj:55)
    clojure.core/first (core.clj:55)
    cemerick.pomegranate.aether$deploy_artifacts.invokeStatic (aether.clj:361)
    cemerick.pomegranate.aether$deploy_artifacts.doInvoke (aether.clj:308)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invokeStatic (core.clj:646)
    clojure.core$apply.invoke (core.clj:641)
    cemerick.pomegranate.aether$deploy.invokeStatic (aether.clj:427)
    cemerick.pomegranate.aether$deploy.doInvoke (aether.clj:391)
    clojure.lang.RestFn.invoke (RestFn.java:619)
    leiningen.deploy$deploy.invokeStatic (deploy.clj:236)
    leiningen.deploy$deploy.invoke (deploy.clj:193)
    clojure.lang.Var.invoke (Var.java:383)
    clojure.lang.AFn.applyToHelper (AFn.java:156)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.core$apply.invokeStatic (core.clj:648)
    clojure.core$apply.invoke (core.clj:641)
    leiningen.core.main$partial_task$fn__1770.doInvoke (main.clj:284)
    clojure.lang.RestFn.applyTo (RestFn.java:139)
    clojure.lang.AFunction$1.doInvoke (AFunction.java:29)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invokeStatic (core.clj:648)
    clojure.core$apply.invoke (core.clj:641)
    leiningen.core.main$apply_task.invokeStatic (main.clj:334)
    leiningen.core.main$apply_task.invoke (main.clj:320)
    leiningen.core.main$resolve_and_apply.invokeStatic (main.clj:340)
    leiningen.core.main$resolve_and_apply.invoke (main.clj:336)
    leiningen.core.main$_main$fn__1843.invoke (main.clj:449)
    leiningen.core.main$_main.invokeStatic (main.clj:439)
    leiningen.core.main$_main.doInvoke (main.clj:436)
    clojure.lang.RestFn.invoke (RestFn.java:421)
    clojure.lang.Var.invoke (Var.java:383)
    clojure.lang.AFn.applyToHelper (AFn.java:156)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.core$apply.invokeStatic (core.clj:646)
    clojure.main$main_opt.invokeStatic (main.clj:314)
    clojure.main$main_opt.invoke (main.clj:310)
    clojure.main$main.invokeStatic (main.clj:421)
    clojure.main$main.doInvoke (main.clj:384)
    clojure.lang.RestFn.invoke (RestFn.java:457)
    clojure.lang.Var.invoke (Var.java:394)
    clojure.lang.AFn.applyToHelper (AFn.java:165)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.main.main (main.java:37)