Skip to content
Please note that GitHub no longer supports Internet Explorer.

We recommend upgrading to the latest Microsoft Edge, Google Chrome, or Firefox.

Learn more
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get rid of sha1 for fixed-output derivations in nixpkgs #77238

Open
grahamc opened this issue Jan 7, 2020 · 6 comments
Open

Get rid of sha1 for fixed-output derivations in nixpkgs #77238

grahamc opened this issue Jan 7, 2020 · 6 comments

Comments

@grahamc
Copy link
Member

@grahamc grahamc commented Jan 7, 2020

Issue description

We're in 2020 and:

Biggest usage of sha1 is files generated for Node packages:

2372	pkgs/development/node-packages/node-packages-v10.nix
1741	pkgs/servers/web-apps/codimd/yarn.nix
1644	pkgs/applications/version-management/gitlab/yarnPkgs.nix
1485	pkgs/servers/monitoring/prometheus/webui-yarndeps.nix
1481	pkgs/servers/gotify/yarndeps.nix
476	pkgs/development/tools/yarn2nix-moretea/yarn2nix/yarn.nix
421	pkgs/applications/networking/cluster/spacegun/node-packages.nix
290	pkgs/servers/rippled/package.nix
213	pkgs/development/compilers/elm/packages/node-packages.nix
189	pkgs/development/web/remarkjs/node-packages.nix
169	pkgs/development/mobile/androidenv/generated/packages.nix
141	pkgs/development/node-packages/node-packages-v12.nix
127	pkgs/applications/networking/instant-messengers/riot/riot-desktop-yarndeps.nix
116	pkgs/servers/matrix-synapse/matrix-appservice-slack/node-packages.nix
88	pkgs/misc/base16-builder/node-packages-generated.nix
74	pkgs/development/mobile/androidenv/generated/addons.nix
73	pkgs/development/node-packages/node-packages-v13.nix
70	pkgs/development/compilers/graalvm/default.nix
66	pkgs/applications/networking/instant-messengers/matrix-recorder/node-packages.nix
48	pkgs/tools/networking/airfield/node-packages.nix
48	pkgs/development/misc/google-clasp/node-packages.nix
36	pkgs/development/mobile/androidenv/generated/system-images-android.nix
33	pkgs/servers/web-apps/cryptpad/node-packages-generated.nix
33	pkgs/development/mobile/androidenv/generated/system-images-google_apis.nix
10	pkgs/development/mobile/androidenv/generated/system-images-android-tv.nix
6	pkgs/tools/package-management/nixui/node-packages.nix
6	pkgs/development/mobile/androidenv/generated/system-images-google_apis_playstore.nix
6	pkgs/development/mobile/androidenv/generated/system-images-android-wear.nix
6	pkgs/development/mobile/androidenv/convertaddons.xsl
4	pkgs/development/mobile/androidenv/generated/system-images-android-wear-cn.nix
2	pkgs/development/mobile/androidenv/convertpackages.xsl
2	pkgs/applications/networking/browsers/mozilla-plugins/google-talk-plugin/default.nix
1	pkgs/tools/typesetting/tex/texlive/default.nix
1	pkgs/development/tools/yarn2nix-moretea/yarn2nix/lib/generateNix.js
1	pkgs/development/tools/unity3d/default.nix
1	pkgs/development/mobile/androidenv/convertsystemimages.xsl
1	pkgs/development/libraries/wxsqliteplus/default.nix
1	pkgs/development/libraries/wxsqlite3/default.nix
1	pkgs/development/haskell-modules/configuration-hackage2nix.yaml
1	pkgs/development/haskell-modules/configuration-common.nix
1	pkgs/build-support/vm/rpm/rpm-closure.pl
1	pkgs/applications/office/grisbi/default.nix
1	pkgs/applications/graphics/gcolor2/default.nix
1	doc/release-notes.xml

Steps to resolve, higher level

  • generator tools should be updated to use better hashes:
    • node2nix
    • yarn2nix
  • generator tools should be updated to print out how they were generated:
    • yarn2nix-moretea.yarn2nix
  • update all the packages which use generated dependency files
  • update all the packages which use sha1 without a generator

Files to address

  • pkgs/applications/graphics/gcolor2/default.nix
  • pkgs/applications/networking/browsers/mozilla-plugins/google-talk-plugin/default.nix
  • pkgs/applications/networking/cluster/spacegun/node-packages.nix
  • pkgs/applications/networking/instant-messengers/matrix-recorder/node-packages.nix
  • pkgs/applications/networking/instant-messengers/riot/riot-desktop-yarndeps.nix
  • pkgs/applications/office/grisbi/default.nix
  • pkgs/applications/version-management/gitlab/yarnPkgs.nix
  • pkgs/build-support/vm/rpm/rpm-closure.pl
  • pkgs/development/compilers/elm/packages/node-packages.nix
  • pkgs/development/compilers/graalvm/default.nix
  • pkgs/development/haskell-modules/configuration-common.nix (false positive)
  • pkgs/development/haskell-modules/configuration-hackage2nix.yaml (false positive)
  • pkgs/development/libraries/wxsqlite3/default.nix
  • pkgs/development/libraries/wxsqliteplus/default.nix
  • pkgs/development/misc/google-clasp/node-packages.nix
  • pkgs/development/mobile/androidenv/convertaddons.xsl
  • pkgs/development/mobile/androidenv/convertpackages.xsl
  • pkgs/development/mobile/androidenv/convertsystemimages.xsl
  • pkgs/development/mobile/androidenv/generated/addons.nix
  • pkgs/development/mobile/androidenv/generated/packages.nix
  • pkgs/development/mobile/androidenv/generated/system-images-android.nix
  • pkgs/development/mobile/androidenv/generated/system-images-android-tv.nix
  • pkgs/development/mobile/androidenv/generated/system-images-android-wear-cn.nix
  • pkgs/development/mobile/androidenv/generated/system-images-android-wear.nix
  • pkgs/development/mobile/androidenv/generated/system-images-google_apis.nix
  • pkgs/development/mobile/androidenv/generated/system-images-google_apis_playstore.nix
  • pkgs/development/node-packages/node-packages-v10.nix
  • pkgs/development/node-packages/node-packages-v12.nix
  • pkgs/development/node-packages/node-packages-v13.nix
  • pkgs/development/tools/unity3d/default.nix
  • pkgs/development/tools/yarn2nix-moretea/yarn2nix/lib/generateNix.js
  • pkgs/development/tools/yarn2nix-moretea/yarn2nix/yarn.nix
  • pkgs/development/web/remarkjs/node-packages.nix
  • pkgs/misc/base16-builder/node-packages-generated.nix
  • pkgs/servers/gotify/yarndeps.nix
  • pkgs/servers/matrix-synapse/matrix-appservice-slack/node-packages.nix
  • pkgs/servers/monitoring/prometheus/webui-yarndeps.nix
  • pkgs/servers/rippled/package.nix
  • pkgs/servers/web-apps/codimd/yarn.nix
  • pkgs/servers/web-apps/cryptpad/node-packages-generated.nix
  • pkgs/tools/networking/airfield/node-packages.nix
  • pkgs/tools/package-management/nixui/node-packages.nix
  • pkgs/tools/typesetting/tex/texlive/default.nix
@grahamc grahamc added this to the 20.03 milestone Jan 7, 2020
@grahamc grahamc changed the title Get rid of sha1 support for fixed-output derivations Get rid of sha1 for fixed-output derivations in nixpkgs Jan 7, 2020
turboMaCk added a commit to turboMaCk/nixpkgs that referenced this issue Jan 7, 2020
related to: NixOS#77238
@turboMaCk turboMaCk mentioned this issue Jan 7, 2020
2 of 10 tasks complete
turboMaCk added a commit to turboMaCk/nixpkgs that referenced this issue Jan 7, 2020
related NixOS#77238
@turboMaCk turboMaCk mentioned this issue Jan 7, 2020
1 of 10 tasks complete
turboMaCk added a commit to turboMaCk/nixpkgs that referenced this issue Jan 7, 2020
related NixOS#77238
turboMaCk added a commit to turboMaCk/nixpkgs that referenced this issue Jan 7, 2020
related NixOS#77238
@edolstra

This comment has been minimized.

Copy link
Member

@edolstra edolstra commented Jan 7, 2020

We can script most of this by using maintainers/scripts/find-tarballs.nix to get all the URLs/hashes, fetching them and then doing s/$sha1/$sha256 on the Nix expression. Or use tarballs.nixos.org to map sha1 hashes to sha512:

$ curl -v https://tarballs.nixos.org/sha1/e410a52dcff3d5c6c3d448b68a026d04ccd744be
location: /sha512/90ae204071cdab3b5e0da28709afa8ac0a5ee13737254cded76a6b8cf323ed2b42390aaa10dbb7935b2aaa57504ceba21306a7eaeea088ab85ee9bdb0a05b530
@cdepillabout

This comment has been minimized.

Copy link
Member

@cdepillabout cdepillabout commented Jan 8, 2020

@grahamc Thanks for looking into this. I think these types of security related fixes are important!

I was wondering how you determined the list of files using sha1?

I wanted to fix the haskell-related files (pkgs/development/haskell-modules/configuration-common.nix and pkgs/development/haskell-modules/configuration-hackage2nix.yaml), so I grepped through the files looking for the strings sha1 and hash, but I couldn't find anything suspicious. Also configuration-hackage2nix.yaml doesn't appear to have any hashes at all.

@grahamc

This comment has been minimized.

Copy link
Member Author

@grahamc grahamc commented Jan 8, 2020

I did a simple search of sha1 =. It has a few false positives, for sure :) Please tick them as done if you find them!

@7c6f434c

This comment has been minimized.

Copy link
Member

@7c6f434c 7c6f434c commented Jan 8, 2020

git grep -l -E '(^| )sha1\> *= *' should catch less false positives, I believe.

Once we clean up all the generators, we can rerun with \<sha1\> *= as a safety check.

@MetaDark

This comment has been minimized.

Copy link
Contributor

@MetaDark MetaDark commented Jan 10, 2020

All the packages in node-packages-v10.nix that use sha1, only do so because they don't list a sha512 hash in the npm registry. Regenerating this file is already slow enough, if we wanted to use sha256 or sha512, node2nix (and yarn2nix) would also have to download the files.

For example, acorn 4.0.13 doesn't have a sha512 hash:

nix-shell -p curl jq --run 'curl http://registry.npmjs.org/acorn/4.0.13 | jq .dist'
{
  "shasum": "105495ae5361d697bd195c825192e1ad7f253787",
  "tarball": "https://registry.npmjs.org/acorn/-/acorn-4.0.13.tgz"
}

but version 7.1.0 does:

nix-shell -p curl jq --run 'curl http://registry.npmjs.org/acorn/7.1.0 | jq .dist'
{
  "integrity": "sha512-kL5CuoXA/dgxlBbVrflsflzQ3PAas7RYZB52NOm/6839iVYJgKMJ3cQJD+t2i5+qFa8h3MDpEOJiS64E8JLnSQ==",
  "shasum": "949d36f2c292535da602283586c2477c57eb2d6c",
  "tarball": "https://registry.npmjs.org/acorn/-/acorn-7.1.0.tgz",
  "fileCount": 11,
  "unpackedSize": 1104477,
  "npm-signature": "-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJdibjuCRA9TVsSAnZWagAAmQgQAIkJPo9i+wdBLLZY8yjT\nHI82RsWU5OvIQxhE6H9qLiEfF57ZSn3LfKVugsEocc84loCiJkkd7vsGsidJ\nRMbkEIK2ENjH1VcUHdHq8PnA/HgnMgHNq6nGg+h6aXLsN/H5SFdyvWedoLXw\n5bwhLbfURV/00bx2himrt/RBNL8SxfowdpQ/ps5+mAWUzLTf7D2+PchYktve\nRJ1lSwpshrgu4AxUdbRmqDEH7+Wm5SutB/EMkQPxQV/sJviFzB8qqFlQopJS\n83RuvIhQb7ZNUQeh5A1+pQocFU+yd227m9tAr69UeES8vinI+0FYVh5GNh0y\nk5MoiQiG3LROtO1RNea0uYGv8W7cU9CK/6MnHbDMUgr+fpEeU/VGItZfOLhB\nTg10PpYCKtQCsF+Z36YLTXgxeWEoRvvKWmJET/MDQzBM/pRzMRpPHiWn6E9T\nl9wt1WLix2qM1jymwaq+0H6hcCtu0ioY7qn4GGeCAfVlBxWYcydNR11Rb++n\nO3oXEqhc8oxKy+hIDHIMDDz/YfCrJ9q7ZzMO9Ie5dzOqeLxty4fbuDeLcD9U\nkbuA+I8dfEFmO2UtRwyx3k9yjX2Tsf2oAC8VOTINN8TXiauyTjWyU//lSKDX\nmUHBDXXZhKWgygi3TMNW7gvmzYuzmDcbkienQeoADvONpCuJNmraVaC33n1h\ncQmj\r\n=blZm\r\n-----END PGP SIGNATURE-----\r\n"
}
dtzWill added a commit to dtzWill/nixpkgs that referenced this issue Jan 11, 2020
related NixOS#77238

(cherry picked from commit 741db8c)
@vcunat

This comment has been minimized.

Copy link
Member

@vcunat vcunat commented Jan 11, 2020

texlive: it has one "sha1 =" string but the code is generic – pkgs/tools/typesetting/tex/texlive/fixedHashes.nix has over 8k sha1 hashes. I haven't looked into complications when generating these, but if I look right, the file's size would increase roughly 460k -> 624k. (Seems acceptable, I suppose, given the circumstances.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.