v8.6.0 -> v8.6.3 breaks bzip2 tarballs

[timbess]

I downgraded to v8.6.0 for now and it works.

pnpm version: v8.6.3

Code to reproduce the issue:

pnpm add https://github.com/pyodide/pyodide/releases/download/0.23.3/pyodide-0.23.3.tar.bz2

Expected behavior:

Successfully installs pyodide from a Github release.

Actual behavior:

it fails before the file is even fully downloaded:

 ERR_PNPM_TARBALL_EXTRACT  Failed to unpack the tarball from "https://github.com/pyodide/pyodide/releases/download/0.23.3/pyodide-0.23.3.tar.bz2": Invalid tar header. Maybe the tar is corrupted or it needs to be gunzipped?pyodide/releases/download/0.23.3/pyodide-0.23.3.tar.bz2: 16.4 kB/206 MB

Additional information:

• node -v prints:
  v20.3.0
• Windows, macOS, or Linux?:
  M2 Mac

[await-ovo]

It looks related to #6642, After replacing decompress-maybe with gunzip-maybe, the bzip2 file does not seem to be decompressed properly.

But I'm not sure if this is a bug or intentional, I tried npm v9 and yarn v2 to install the tarball above and it didn't work either.

BTW, you can use pnpm add https://registry.npmjs.org/pyodide/-/pyodide-0.23.3.tgz to avoid the error reported above.

[zkochan]

I don't know what to do about this. I never knew bzip2 was supported. For the npm registry we only really need tgz support. This seems like an extremely rarely used feature.

[timbess]

It's a really useful feature for me since pyodide doesn't release this particular package on NPM. Also it's a breaking change on a bugfix release? Idk how pnpm does versioning or if that matters for y'all.

[zkochan]

We had to remove decompress-maybe from our dependencies because it had a breaking change released as a non-major version bump. So pnpm started to fail.

I was surprised when I have found out that pnpm can unpack something else as it was never a feature that we added knowingly.

If it doesn't make installation slower, I guess we can add it back but we cannot use decompress-maybe as it now fails if there is no ZSTD installed on the system. Or we can use an older version maybe.

cc @pnpm/collaborators

[await-ovo]

I think we can keep this issue and see if more people need this feature.

If we want to do it later, we can just use bzip2-maybe or keep the old version of decompress-maybe

[kjxbyz]

node: 16.20.1
pnpm: 8.6.12

 WARN  GET https://registry.npmjs.org/tsconfig-paths-webpack-plugin/-/tsconfig-paths-webpack-plugin-4.1.0.tgz error (ERR_PNPM_TARBALL_EXTRACT). Will retry in 10 seconds. 2 retries left.
 WARN  GET https://registry.npmjs.org/@swc/core-darwin-x64/-/core-darwin-x64-1.3.75.tgz error (ERR_PNPM_TARBALL_EXTRACT). Will retry in 10 seconds. 2 retries left.
Downloading registry.npmjs.org/@swc/core-darwin-x64/1.3.75: 13.63 MB/13.63 MB, done
 WARN  deprecated @apollo/server-plugin-landing-page-graphql-playground@4.0.0: The use of GraphQL Playground in Apollo Server was supported in previous versions, but this is no longer the case as of December 31, 2022. This package exists for v4 migration purposes only. We do not intend to resolve security issues or other bugs with this package if they arise, so please migrate away from this to [Apollo Server's default Explorer](https://www.apollographql.com/docs/apollo-server/api/plugin/landing-pages) as soon as possible.
 WARN  deprecated subscriptions-transport-ws@0.11.0: The `subscriptions-transport-ws` package is no longer maintained. We recommend you use `graphql-ws` instead. For help migrating Apollo software to `graphql-ws`, see https://www.apollographql.com/docs/apollo-server/data/subscriptions/#switching-from-subscriptions-transport-ws    For general help using `graphql-ws`, see https://github.com/enisdenjo/graphql-ws/blob/master/README.md
 WARN  GET https://registry.npmjs.org/swagger-ui-dist/-/swagger-ui-dist-5.3.1.tgz error (ERR_PNPM_TARBALL_EXTRACT). Will retry in 10 seconds. 2 retries left.
 WARN  GET https://registry.npmjs.org/tsconfig-paths-webpack-plugin/-/tsconfig-paths-webpack-plugin-4.1.0.tgz error (ERR_PNPM_TARBALL_EXTRACT). Will retry in 1 minute. 1 retries left.
 WARN  GET https://registry.npmjs.org/@swc/core-darwin-x64/-/core-darwin-x64-1.3.75.tgz error (ERR_PNPM_TARBALL_EXTRACT). Will retry in 1 minute. 1 retries left.
Packages: +868
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Packages are hard linked from the content-addressable store to the virtual store.
  Content-addressable store is at: /Users/ying/Library/pnpm/store/v3
  Virtual store is at:             node_modules/.pnpm
 WARN  GET https://registry.npmjs.org/swagger-ui-dist/-/swagger-ui-dist-5.3.1.tgz error (ERR_PNPM_TARBALL_EXTRACT). Will retry in 1 minute. 1 retries left.
 ERR_PNPM_TARBALL_EXTRACT  Failed to unpack the tarball from "https://registry.npmjs.org/tsconfig-paths-webpack-plugin/-/tsconfig-paths-webpack-plugin-4.1.0.tgz": incorrect data check
Progress: resolved 876, reused 864, downloaded 0, added 865

[await-ovo]

node: 16.20.1
pnpm: 8.6.12

 WARN  GET https://registry.npmjs.org/tsconfig-paths-webpack-plugin/-/tsconfig-paths-webpack-plugin-4.1.0.tgz error (ERR_PNPM_TARBALL_EXTRACT). Will retry in 10 seconds. 2 retries left.
 WARN  GET https://registry.npmjs.org/@swc/core-darwin-x64/-/core-darwin-x64-1.3.75.tgz error (ERR_PNPM_TARBALL_EXTRACT). Will retry in 10 seconds. 2 retries left.
Downloading registry.npmjs.org/@swc/core-darwin-x64/1.3.75: 13.63 MB/13.63 MB, done
 WARN  deprecated @apollo/server-plugin-landing-page-graphql-playground@4.0.0: The use of GraphQL Playground in Apollo Server was supported in previous versions, but this is no longer the case as of December 31, 2022. This package exists for v4 migration purposes only. We do not intend to resolve security issues or other bugs with this package if they arise, so please migrate away from this to [Apollo Server's default Explorer](https://www.apollographql.com/docs/apollo-server/api/plugin/landing-pages) as soon as possible.
 WARN  deprecated subscriptions-transport-ws@0.11.0: The `subscriptions-transport-ws` package is no longer maintained. We recommend you use `graphql-ws` instead. For help migrating Apollo software to `graphql-ws`, see https://www.apollographql.com/docs/apollo-server/data/subscriptions/#switching-from-subscriptions-transport-ws    For general help using `graphql-ws`, see https://github.com/enisdenjo/graphql-ws/blob/master/README.md
 WARN  GET https://registry.npmjs.org/swagger-ui-dist/-/swagger-ui-dist-5.3.1.tgz error (ERR_PNPM_TARBALL_EXTRACT). Will retry in 10 seconds. 2 retries left.
 WARN  GET https://registry.npmjs.org/tsconfig-paths-webpack-plugin/-/tsconfig-paths-webpack-plugin-4.1.0.tgz error (ERR_PNPM_TARBALL_EXTRACT). Will retry in 1 minute. 1 retries left.
 WARN  GET https://registry.npmjs.org/@swc/core-darwin-x64/-/core-darwin-x64-1.3.75.tgz error (ERR_PNPM_TARBALL_EXTRACT). Will retry in 1 minute. 1 retries left.
Packages: +868
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Packages are hard linked from the content-addressable store to the virtual store.
  Content-addressable store is at: /Users/ying/Library/pnpm/store/v3
  Virtual store is at:             node_modules/.pnpm
 WARN  GET https://registry.npmjs.org/swagger-ui-dist/-/swagger-ui-dist-5.3.1.tgz error (ERR_PNPM_TARBALL_EXTRACT). Will retry in 1 minute. 1 retries left.
 ERR_PNPM_TARBALL_EXTRACT  Failed to unpack the tarball from "https://registry.npmjs.org/tsconfig-paths-webpack-plugin/-/tsconfig-paths-webpack-plugin-4.1.0.tgz": incorrect data check
Progress: resolved 876, reused 864, downloaded 0, added 865

Could you share a package.json to reproduce this problem ?

[zkochan]

It is not related to this issue.

[mnlbox]

I'm also getting the same error as below after updating the PNPM to 8.7.0:

WARN  GET https://registry.npmjs.org/@yarnpkg/parsers/-/parsers-3.0.0-rc.48.1.tgz error (ERR_PNPM_TARBALL_EXTRACT). Will retry in 10 seconds. 2 retries left.

Should I downgrade until a fix?

[zkochan]

@mnlbox no, your issue is not related to this because your issue is with a tgz archive.

In #6850 we have changed how we extract packages and used @dmichon-msft's tarball parser. Looks like there are some edge cases that are not handled by the new extractor.

[zkochan]

It doesn't look like a lot of people need this feature. If we'd ever want to support it, the easiest way would be to create a new custom tarball fetcher specifically for tarballs using bzip2. However, right now it seems like this would be not used by many, so I wouldn't bother adding complexity to the codebase.

[crypt0grapher]

I'm experiencing that when I'm trying to install packages from GitHub commits

pnpm add https://github.com/KevinVandy/mantine-react-table/tree/mantine-7.8/packages/mantine-react-table

What I'm getting is

WARN  GET https://github.com/KevinVandy/mantine-react-table/tree/mantine-7.8/packages/mantine-react-table error (ERR_PNPM_TARBALL_EXTRACT). Will retry in 10 seconds. 2 retries left.