General error when creating mulled containers

[ggabernet]

Hi,

I'm hitting an error when creating mulled containers that does not seem to be related to the specified dependencies, as it happens after a successful environment is built and all of the latest PRs are getting it, even the automated ones.

Here is the relevant part of the error message:

[Dec 18 13:16:16] SOUT Exploding layer: sha256:aa44502a478a5d41773e70f3357e5ef57a9365721e9d3c10627c87e5a207fb59.tar.gz
[Dec 18 13:16:16] SERR 
[Dec 18 13:16:16] SERR gzip: /root/.singularity/docker/sha256:aa44502a478a5d41773e70f3357e5ef57a9365721e9d3c10627c87e5a207fb59.tar.gz: not in gzip format
[Dec 18 13:16:16] SERR tar: This does not look like a tar archive
[Dec 18 13:16:16] SERR tar: Exiting with failure status due to previous errors
[Dec 18 13:16:16] SOUT Cleaning up...
[Dec 18 13:16:16] ERRO Task processing failed: Unexpected exit code [2] of container [a6c8df8d6969 step-fd70b3ce91], container preserved
..
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.15/x64/bin/mulled-build-files", line 8, in <module>
    sys.exit(main())
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/galaxy/tool_util/deps/mulled/mulled_build_files.py", line 42, in main
    ret = mull_targets(
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/galaxy/tool_util/deps/mulled/mulled_build.py", line 286, in mull_targets
    ret = involucro_context.exec_command(involucro_args)
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/site-packages/galaxy/tool_util/deps/mulled/mulled_build.py", line 344, in exec_command
    shutil.rmtree('./build')
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/shutil.py", line 718, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/shutil.py", line 655, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/shutil.py", line 655, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/shutil.py", line 675, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/shutil.py", line 673, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'ncurses-6.3-h27087fc_1.json'
Error: Process completed with exit code 1.

Here is the full error message:
https://github.com/BioContainers/multi-package-containers/pull/2439/checks

Several of the latest PRs are getting it including:
#2441
#2439
#2438

[bgruening]

Jupp, Singularity has problems with pulling the container. I assume that a recent Singularity version broke something but I had no time to look deeper into it.

[ggabernet]

Hi, as quite some people depend on building mulled containers this way, would it maybe be an idea to pin the old Singularity version until the issue is solved?

[tamuanand]

Hi @ggabernet and @bgruening

• I get a similar error when building mulled containers
• bbmap,seqkit,pigz,csvtk multi-package #2449 (comment)

[bgruening]

I fixed that upstream, just need to get the fix deployed here as well. Something changed in Singularity. Sorry, just need a bit more time.

[tamuanand]

Hi @bgruening

Thanks a lot for looking into this.

Quick question: Once this is fixed, will the failed tests run automatically (OR) does the PR submitter need to do something to initiate the tests and the container build?

Thanks once again.

[bgruening]

I will rerun the tests for you and if it turns green, merge it. Sorry for the inconvenience.

[ggabernet]

Hi @bgruening , thanks a lot for fixing this!

[bgruening]

It should be fixed on master. There is now a small backlog of containers which I will try to resolve. If your PR is still failing, its likely that its the fault of the PR. Please wait another 24h or try to retrigger your PR on your own.

Thanks for your patience.