Web based package diffs

[joladev]

Hi! I wanted to bring this issue up for discussion. I'll try to describe the importance of package diffs, existing solutions, some possible architectures and some possible solutions that might fit hex.pm. By web-based diffs I mean highlighted outputs from git diff in a browser, with shareable links.

We first started seeing npm packages get hijacked, but recently RubyGems has been having issues (rest_client, strong_password, many others). By hijacking I mean the the type of attack where someone gets access to the credentials of the author of a package and uploads malicious versions. Some examples of scenarios:

1. You merge the new updates and deploy, now you're running infected code in production. They can then mine cryptocurrency or inject HTTP handlers that respond to certain payloads, even giving access to servers and databases.
2. You do automatic bumps in CI. When you run the tests any malicious dependency can read your code or env variables and send them home. Potential leaks: Third party secrets (like AWS credentials), entire source code of your application

A better workflow for updating dependencies

If we can make it easy and painless to audit dependency updates, even if it just means scrolling through the diff, we can close this avenue of attack partially or completely.

Some services already exist for this, like one for npm and one for RubyGems, and I've created a POC for Hex at https://diff.jola.dev.

Architectural problems

I see two broad directions to take with implementing this

1. Create diffs on demand
2. Create and store all diffs somewhere ahead of time, generate new diffs whenever a package is updated

Both have issues and are non-trivial to implement, depending on how reliable we want it to be.

As for generating diffs, the most straightforward way seems to be using the one that mix hex.package diff does, which is download the tarballs, unpack them, diff, and then remove leftover files. Potentially it might be possible to generate diffs from every incremental change, eg 0.1.0+0.2.0, 0.2.0+0.3.0, etc, and then merge the diffs, but I don't know if that's really better.

General thoughts

The two directions I suggested are not mutually exclusive, we might want a combination of features.

All diffs are not equal, from an auditing viewpoint. Checking the last few versions is useful, but diffing phoenix 0.1.0 with 1.4.10 is not. We probably want to limit the number of steps supported. Attacks are getting more sophisticated, where one malicious release is made, quickly followed by 1-2 innocuous ones to hide it. It might be a good idea to make it semver aware.

Depending on the direction we go, it might need to be hardened with rate limiting, size limits etc.

On-demand

This is how https://diff.jola.dev works, it generates and caches diffs in ETS. This works surprisingly well, didn't really see any load issues. It should probably be based on an LRU cache to limit memory use.

Pre-generated

This would probably involve building a secondary service that generates diffs and stores them somewhere. Whenever a new version is released it would automatically generate diffs for it. Then the frontend can lookup diffs and no work is done inline.

I'm experimenting with this approach, but assuming we want all possible diffs, the storage required would be huge. UPDATE: I wrote some code to generate all possible diffs for the registry, let it run for an hour or so. It generated something like 85K diffs for 1600 packages. Each diff was compressed, but it still used about 4GB of disk space.

Displaying diffs

There are ready made solutions for making pretty HTML from a git diff, like https://diff2html.xyz/, that can either run from the shell or in the browser. It seems to handle reasonably large diffs even in the browser, there's no lag after it finishes rendering the generated HTML, but rendering it can take a second or two in extreme cases (on my machine). I expect it to work less well on a mobile phone. Arguably we shouldn't expend effort support mobile phones.

We could reduce the work in the browser by generating the HTML on the server side, but of course that's much larger.

API

Finally, there's the question of how we expose this. Here's some potential directions

• API endpoint for diffs, GET /diff/name/from/to, returning a text blob
• Web endpoint for diffs
• A UI for selecting packages and versions, like https://diff.jola.dev
• Adding links to the last few diffs to a package page

A suggestion for a first approach

This issue has a lot of description of context, and it might be hard to agree on a way forward, so I want to suggest a limited implementation that we can build on.

• Generate diffs on demand, with caching
• Create a web endpoint that displays HTML formatted syntax highlighted diffs
• Add links to latest diffs in each package page (eg "see what changed since the previous version")
• Support all versions of a package, but only a comparison within 5 versions

With the last bullet I am referring to the issue of generating diffs for 0.1.0..1.4.10. For auditing purposes we generally only need to diff each version with the preceding one. I just suggest 5 versions to keep some flexibility, but we could do fewer.

This seems to have a lot of potential and I'm very interested in feedback!

Last thoughts

This is not a trivial project and I expect some discussion here. It might be good to focus on the smallest possible implementation that would still be useful, to get a reasonable scope. It might also be that we don't feel this auditing functionality should be part of hex.pm, but rather could be left to a separate service, like the one I set up.

I also expect this issue to bring new ideas and creative solutions. This is all part of us as a community putting effort into finding better ways to work securely within the ecosystem.

[JonRowe]

Theres a great thread about this and similar issues for rubygems which is the Ruby equivalent of hex.pm rubygems/rubygems.org#2101

[supersimple]

@joladev First off - thank you for this. Your POC is very good.

but diffing phoenix 0.1.0 with 1.4.10 is not

Why do you feel this is not useful? I would hope no one is really trying to do this upgrade, but if they were I would feel like they would want all of the diffs.

A UI for selecting packages and versions, like https://diff.jola.dev

I like this look. We could always iterate on it with the help of the community.

Based on the traffic, I might suggest we run them all on demand, but cache them once they are run.
I suspect there are going to be a lot of demand for Ecto diffs, but a lot of libs with limited usage might never be diffed via website. No need to pre-calculate those diffs IMO. In addition, we could remove them from the cache after a certain amount of time not being accessed (in cases like phx1.0 -> phx1.1 where there is probably near-zero demand after a certain amount of time.)

[JonRowe]

Based on the traffic, I might suggest we run them all on demand

Dependant on budget / technology choices, its possible to do this with AWS S3 static hosting and a "Redirect when content missing" rule, where the content can then be generated and returned for the request and cached in S3 for later retrieval.

[supersimple]

@JonRowe yes. we use GCP, but something similar exists.

[ericmj]

Thank you for starting the discussion and building https://diff.jola.dev. Easy package diffing is something everyone on the Hex team wants so we appreciate that you got the ball rolling on this.

Ideally we would want it to be part of the "hexpm group of services" but a separate, well maintained service by the community would of course also work. Assuming it is part of the hexpm group it could be part of the the hexpm project (this repository) or built as a separate service, depending on how we chose to implement it. A separate service would add more complexity, but it has some merits if we want to add features such as caching or precomputing of diffs.

Like you suggest I think we should start with a minimal set of features. From your list of potential API directions I think "A UI for selecting packages and versions, like https://diff.jola.dev" + "Web endpoint for diffs" gives the most value to users so starting there would make sense. An API that returns a text blob would also be useful and could simplify the client implementation, but would add less immediate value. I think we can try by doing the diffing on-demand and then start optimizing if we notice that it is too resource consuming.

Something else that has come up in the past when we have talked about package diffing is browsing package file contents. Right now it is difficult to see the actual contents of a package. Looking at the github repository is not safe in case of a bad actor so you need to run mix deps.get or use the newly added mix hex.package fetch PACKAGE VERSION --unpack task. A website where you can browse the contents would help a lot to verify packages. A package browser cannot work on-demand though, we can't see the full package contents to clients and downloading and unpacking the full tarball to send one file does not make sense I think. So in that case a separate service that caches the contents or dumps the unpacked contents into an S3 bucket on new publishes would be required.

We should definitely build the differ before the browser but I just wanted to mention that it is something we want to have eventually and since they may share features it could be good to keep in the back of our heads as we work on the differ.

[ericmj]

Closing in favor of #894.