[bug] Large conan package gets downloaded to a wrong subfolder in windows

[randomcontainers]

Describe the bug

When downloading a large (>10mb) conan package on my windows machine, I stumbled upon following problem:

C:\Users\user>conan download -r conan-remote linux-headers-6.6-device/6.6.37-device
Downloading recipes in 4 parallel threads
Downloading recipe 'linux-headers-6.6-device/6.6.37-device#0894beb9f841cbc60af37dcda41a161e'
Downloading 'linux-headers-6.6-device/6.6.37-device' sources
Downloading binary packages in 4 parallel threads
Downloading package 'linux-headers-6.6-device/6.6.37-device#0894beb9f841cbc60af37dcda41a161e:da39a3ee5e6b4b0d3255bfef95601890afd80709#3588b62173715dfee3e472a622fb6635'
linux-headers-6.6-device/6.6.37-device: Retrieving package da39a3ee5e6b4b0d3255bfef95601890afd80709 from remote 'conan-remote'
linux-headers-6.6-device/6.6.37-device: Downloading 11.6MB conan_package.tgz
linux-headers-6.6-device/6.6.37-device: Decompressing 11.6MB conan_package.tgz
linux-headers-6.6-device/6.6.37-device: ERROR: Exception while getting package: da39a3ee5e6b4b0d3255bfef95601890afd80709
linux-headers-6.6-device/6.6.37-device: ERROR: Exception: <class 'conans.errors.ConanException'> Error while extracting downloaded file 'C:\Users\user.conan2\p\linux7f4334d1b9228\d\conan_package.tgz' to C:\Users\user.conan2\p\linux7f4334d1b9228\p
[Errno 2] No such file or directory: 'C:\Users\user\.conan2\p\linux7f4334d1b9228\p\include\soc\arc\aux.h'
Folder removed
ERROR: Error while extracting downloaded file 'C:\Users\user.conan2\p\linux7f4334d1b9228\d\conan_package.tgz' to C:\Users\user.conan2\p\linux7f4334d1b9228\p
[Errno 2] No such file or directory: 'C:\Users\user\.conan2\p\linux7f4334d1b9228\p\include\soc\arc\aux.h'
Folder removed

This is on conan version 2.5.0
When I try look at the .conan2/p directory, I can see that there is only 1 new package with a different name than expected:

Directory of C:\Users\user.conan2\p

07/16/2024 08:07 AM

.
07/16/2024 08:07 AM ..
07/16/2024 08:07 AM 28,672 cache.sqlite3
07/16/2024 08:07 AM linux3b28e86b22ffd
1 File(s) 28,672 bytes
3 Dir(s) 87,820,476,416 bytes free

I can browse through the directory and see that it indeed belongs to my conan package, although the p dir seems to be missing:

Directory of C:\Users\user.conan2\p\linux3b28e86b22ffd

07/16/2024 08:07 AM

.
07/16/2024 08:07 AM ..
07/16/2024 08:07 AM d
07/16/2024 08:07 AM e
07/16/2024 08:07 AM es
0 File(s) 0 bytes
5 Dir(s) 87,817,330,688 bytes free

Directory of C:\Users\user.conan2\p\linux3b28e86b22ffd\d

07/16/2024 08:07 AM

.
07/16/2024 08:07 AM ..
07/16/2024 08:07 AM 8,619,899 conan_sources.tgz
1 File(s) 8,619,899 bytes
2 Dir(s) 87,814,180,864 bytes free

Directory of C:\Users\user.conan2\p\linux3b28e86b22ffd\e

07/16/2024 08:07 AM

.
07/16/2024 08:07 AM ..
07/16/2024 08:07 AM 2,104 conanfile.py
07/16/2024 08:07 AM 153 conanmanifest.txt
2 File(s) 2,257 bytes
2 Dir(s) 87,822,680,064 bytes free

Directory of C:\Users\user.conan2\p\linux3b28e86b22ffd\es

07/16/2024 08:07 AM

.
07/16/2024 08:07 AM ..
07/12/2024 10:40 AM 8,616,988 linux-headers-6.6-device_6.6.37-device_arm64.deb
1 File(s) 8,616,988 bytes
2 Dir(s) 87,821,041,664 bytes free

As you can see the "linux7f4334d1b9228" directory, that is expected by conan download, does not exist.
When I download smaller packages, it works without a problem.
Downloading this package on a linux machine works. (the linux7f4334d1b9228 dir becomes available after downloading).

I didn't try if this only happens with this package or also with other large packages on windows.

How to reproduce it

I am unsure if this is reproducible.

use conan download to download a large conan package (>10mb) on a windows machine.
There should be an "Error while extracting"

[memsharded]

Hi @randomcontainers

Thanks for your report.

A couple of quick questions:

• How large is really the package? 10mb is nothing, those are very common packages in Conan, shouldn't be a problem. We have seen in the past some issues under some special infra (google cdn) for packages >1 Gb
• What happens if you try to download that package in a Linux machine, does it works well?
• One possible issue is that such package has some special files in it, like symlinks, not supported in Windows, and then it might indeed fail under windows. Can you try downloading the conan_package.tgz from the server manually and unzipping it with other tools?

[randomcontainers]

Hi,
thanks for your quick response!

How large is really the package? 10mb is nothing, those are very common packages in Conan, shouldn't be a problem. We have seen in the past some issues under some special infra (google cdn) for packages >1 Gb

The package is 11mb in size. I called it a large file because of this line: (big_file)

What happens if you try to download that package in a Linux machine, does it works well?

Yes, on a Linux machine it works without an issue. The difference is that on Linux, after using conan download two new directories are created in the .conan/p directory: linux3b28e86b22ffd and linux7f4334d1b9228. In Windows only linux3b28e86b22ffd is created but it still expects linux7f4334d1b9228 which causes the error.
Also the contents of the linux7f4334d1b9228 directory are different, it contains the actual package whereas linux3b28e86b22ffd contains a debian package.

One possible issue is that such package has some special files in it, like symlinks, not supported in Windows, and then it might indeed fail under windows. Can you try downloading the conan_package.tgz from the server manually and unzipping it with other tools?

This might actually be the issue. When I unzip the package in windows, I get indeed errors on which I can skip or cancel the unzipping.
Does conan cancel the unzip operation in this case?

[memsharded]

Yes, on a Linux machine it works without an issue. The difference is that on Linux, after using conan download two new directories are created in the .conan/p directory: linux3b28e86b22ffd and linux7f4334d1b9228. In Windows only linux3b28e86b22ffd is created but it still expects linux7f4334d1b9228 which causes the error.

Yes, this is expected, there are 2 folders involved:

• One folder containing the "recipe" part (subfolders e=export, containing the recipe, es=export_sources containing exported sources, etc)
• Another folder (one per different package binary with different package_id) containing the actual artifacts, (subfolder b=build if built from source locally, subfolder p=package with the artifacts), like include headers and compiled static or shared libraries. This folder is also obtained by unzipping the conan_package.tgz

If the unzipping of conan_package.tgz fails, Conan will try to remove the second folder, to not leave a trailing package in a bad state. This is the reason why if it fails you will only see the first folder but not the second one.

[randomcontainers]

Thank you for the explanation!

[memsharded]

Thanks to you for the feedback!

I have been thinking about this, just in case it would be possible some workaround, but it doesn't seem possible to do anything from Conan side.

Maybe it might be possible to enable symlinks (developer mode) in Windows?