(Done) May Meetup: Secure Clojure Supply Chains

[frenchy64]

👍 to RSVP

https://madclj.com/

Wednesday, May 14th, 2025
6:30 PM to 9:00 PM CT

StartingBlock Madison
821 E Washington Ave
-!!->> 3rd floor ("Pop" Conference Room) <<-!!-
Madison, WI

What are ways Clojure programs can be exploited via supply chain attacks? Are jars always what they seem? When is a git SHA not enough? What can we do to detect and prevent attacks?

Let's brainstorm some ideas.

• attack vectors?
• who should we "trust"?
• what makes you nervous about adding 3rd party Clojure dependencies?
• build-time injection https://en.wikipedia.org/wiki/XZ_Utils_backdoor
• https://reproducible-builds.org/
• https://www.qubes-os.org/news/2016/05/30/build-security/
• https://inversegravity.net/2019/web-of-trust-dead/
• https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work
• jank simply ignores .o files in jars. can we ignore untrusted AOT compiled Clojure files?
• xz-style backdoor in bb
• https://clojure.org/releases/download_key

Let's try and design and prototype some detection tools during the meetup.

[frenchy64]

@madclj/meetup May meetup: I've been interested in supply-chain attacks for a few years now. I think Clojure has a ways to go in protecting from them in practice. It would be great to get everyone's take on it and maybe we can prototype some ideas.

[sivakusayan]

I won't be able to come next week as I'll be at a concert in Milwaukee, hope it's fun though!

[brunchboy]

Chris and I will be near Las Vegas, getting ready to attend Electric Daisy Carnival, so I will have to miss this month as well.

[djwhitt]

Interested in the topic, but also can't make it today.

[frenchy64]

Thanks everyone, makes organizing food much easier!

[ericnormand]

Will be there with the girls.

[frenchy64]

Note that the meetup today is on the 3rd floor in the "Pop" conference room. StartingBlock is using the 2nd floor classroom for storage after yielding some office space back to the owners.

I'll try and put a sign on the door.

[dpetranek]

I'm waiting outside if someone could let me in 😁

[frenchy64]

Omw!

[frenchy64]

Our discussion was split in two parts. The first part we considered attack vectors (and defenses) during the development lifecycle of adding a new feature to some Clojure application.

Time roughly goes from left-to-right, with the first phase being development on a dev machine, second being CI testing and deployment to an integration environment, and third being a deployment to production.

From left-to-right:

• the dev machine could have installed malicious CLI tools
  • hastily copy-pasted a sha from a malicious source
  • Q: is merely installing a malicious tool a dangerous operation?
• phished looking for libraries
  • attacker forks old library
    • actually improves library at first
    • socially engineer devs to try/depend on it
    • could target original library dev's computer for deploy keys
  • malicious search engine result
  • attack clojars infrastructure to target particular dependees
• a dep could include malicious AOT compiled Clojure code that shadows clean source files
• dev uses Clojure CLI's ability to automatically resolve a SHA from a tag
  • tag could have mutated
• dev uses short sha for git dep and git repo mutates what the short sha resolves to
• during CI tests, code could call home
  • macros might be a more obfuscated attack vector
• in GitHub Actions, we trust the setup-clojure Action
  • we happen to work with the dev
  • but what if his computer was attacked?
  • no protected branches or PR approvals
• pinning an Actions dep to a commit does not pin its dependencies
  • some users now transitively rewrite Actions to pin all dependencies to commits
• man in the middle attacks if using insecure protocol for Maven repositories such as http
• shadowing attacks across maven repos
  • e.g., clojars shadowing Maven Central (addressed by clojars but not in general)
• Maven repositories could mutating jars
  • must trust them or save checksums
  • don't need to trust GitHub when using git dependencies, git sha is not tamperable (to the same degree)
• poisoned CI cache shared between integration and production builds
• construction of uberjar triggers change to uberjar itself
  • like xz attack

Second half we considered what it would concretely mean for Clojure projects to be "more reproducible", why we would even want reproducibility, and where reproducible builds for Clojure stand in May 2025.

On the left we have a measure of "Build Reproducibility", where at the bottom we can reproduce 0% of our Clojure project from source (perhaps we were just handed a native binary), and at the top we can completely recreate our Clojure project from source, bit-for-bit.

On the far right, we have an inverse measure of "Trust" that we are required to have. On the bottom we must 100% trust all parties involved in creating the binary (the diagram lines up horizonally, so 100% trust == 0% reproducible builds). At the top (0%) we don't need to trust anyone because our builds are 100% reproducible. Measure of Trust needed falls much slower than adding reproducibility.

In the middle we have four more measures, from left to right:

1. effort required to make builds this % reproducible
   • could be drastically different at different stages
   • debian's final 20% are the hardest
   • hard to get started
2. Auditability
   • as build is more reproducible, auditability of also improves
   • better auditability helps find zero-day exploits quicker by eliminating impossible scenarios
3. Attack detection time
   • the increased auditability leads to a decrease in attack detection time.
   • 0% reproducible == 1 month to detect attack
   • 100% reproducible == 1 minute
4. Value of zero-day exploits
   • presumably, exploits are more expensive the longer they are predicted to be unpatched
   • value of zero-day exploits goes down as reproducibility goes up
   • 0% reproducible == $1m exploit (lasts for 1 month)
   • 100% reproducible == $1 exploit (lasts for 1 minute)

On the left, we sketched what different % reproducible Clojure projects might look like.

• 0% reproducible
  • no access to source code, just have a binary
  • 100% trust required
    • must trust everyone
• 5% reproducible
  • access to your own source code, but cannot reproduce dependencies
  • baseline for most Clojure applications
  • 95% trust required
    • don't need to trust that your source code was used to build the binary
• 6% reproducible
  • thin main class for uberjar is compiled reproducibly
  • perhaps not as low hanging fruit, but easier than some things above it
• 7.5% reproducible
  • source code for direct clojure deps is available
  • no AOT-compiled sources
  • Clojure CLI supports this via git deps
    • change all dependencies in your deps.edn to git deps
  • trust
    • don't need to trust that dep corresponds to git commit, since git sha's are not tamperable
    • however it would be better to audit the commit in some way
      • read the source code, cross-check release announcement, commit is on release branch that cannot be shadowed by forks
• 10% reproducible
  • source code for all transitive Clojure source code available
    • all corresponding to git sha's
  • not well supported, but possible with Clojure CLI
    • manually add git deps for all transitive deps
• 15% reproducible
  • verified Clojure CLI/Lein build tools are built reproducibly
• 20% reproducible
  • Java jar dependencies are reproducible
  • all class files are reproducible from their source files
• 25% reproducible
  • installed JVM is reproducible from its source code
  • probably already the case on many distros.
• 80% reproducible
  • Debian is here

[frenchy64]

@devn I was wrong about :git/tag, see these tests for the specific behavior.