[BUG] put.stream() with integrity check might not work

[ilg-ul]

What / Why

Apparently put.stream() with integrity checks leaves the cache in an inconsistent state, and all keys inserted with this procedure are purged by the next verify().

I tried to find some use cases of put.stream(), but I could not find anything in npm, nor any tests here, so, although I don't rule out an improper use case in my code, I guess we are facing a small bug that slipped due to incomplete testing.

When

Always.

How

Current Behavior

I'm using put.stream() to add large archives to the cache, and I have the expected sha256 digest for each archive.

If I set opts.algorithms = ['sha256'] and I check the resulting integrity[hashAlgorithm][0].source I get the expected digest, and everything is fine.

If I set opts.integrity to the expected digest, the call seems ok too, it does not throw any error, the archive is saved in the cache and decompressing it is fine.

However, if I run a verify(), the associated entries are removed, as there would be something wrong with them.

Steps to Reproduce

• run a put.stream() with opts.algorithms = ['sha256']
• note the resulting integrity digest
• run a verify()
• run a ls(); the recently added key is there
• run a put.stream() with opts.integrity set to the digest
• note that the call seems ok, it returns the integrity result, which matches the input
• run a verify()
• run a ls(); the key is no longer in the cache

Note: using sha256 might not be relevant, but I mentioned it to match my test.

Expected Behavior

Running verify() after put.stream() with integrity checks should not remove the entries from the cache.

Who

• n/a

References

• n/a

[ilg-ul]

I tried to update my tool that uses cacache to 15.x and it looks like the issue is still present, so I'm currently stuck at 12.x.

Is there anything I can do to help you fix the problem?

[ilg-ul]

I did some tests and the stream interface is no longer functional starting with 13.0.0. In other words, the same code that works with 12.x fails with 13.0.

My current workaround was to use an internal buffer, but since the files that I need to install are large, like 150 MB or more, this is not an optimal solution, especially when running on small systems, like Raspberry Pis.

[ilg-ul]

The internal buffer workaround is far from great, for example it makes AppVeyor tests fail with out of memory:

Adding '@xpack-dev-tools/riscv-none-embed-gcc@8.3.0-1.2.1' to repository as 'C:\Users\appveyor\AppData\Roaming\xPacks\@xpack-dev-tools\riscv-none-embed-gcc\8.3.0-1.2.1'...
Downloading https://github.com/xpack-dev-tools/riscv-none-embed-gcc-xpack/releases/download/v8.3.0-1.2/xpack-riscv-none-embed-gcc-8.3.0-1.2-win32-x32.zip...
Extracting 'xpack-riscv-none-embed-gcc-8.3.0-1.2-win32-x32.zip'...
FATAL ERROR: Committing semi space failed. Allocation failed - process out of memory
1: 011A785E
2: 015B1903
3: 015B0414
4: 015AFFEC

Is there anything that can be done to fix stream support in cacache?

[ilg-ul]

It might be possible that the issue was introduced with f4c0962.