Files copied inside singularity not found (intel_psxe building block) #341

VI-gha · 2021-02-21T09:12:57Z

The intel_psxe building block installs Intel Parallel Studio XE and requires a license file and a tarball:

base image

Stage0 += baseimage(image='ubuntu:{}'.format(18.04), _as='build')

Intel PSXE

#Stage0 += intel_psxe(eula=True, license='license.lic', tarball='parallel_studio_xe_2020_update4_cluster_edition.tgz')

When building a singularity container these files are apparently copied into a temporary folder (whose name changes every time):

INFO: Copying parallel_studio_xe_2020_update4_cluster_edition.tgz to /tmp/build-temp-795513904/rootfs/var/tmp/parallel_studio_xe_2020_update4_cluster_edition.tgz
INFO: Copying license.lic to /tmp/build-temp-795513904/rootfs/var/tmp/license.lic

which means that the build fails when they are not found:

tar -x -f /var/tmp/parallel_studio_xe_2020_update4_cluster_edition.tgz -C /var/tmp -z
tar (child): /var/tmp/parallel_studio_xe_2020_update4_cluster_edition.tgz: Cannot open: No such file or directory

I tried to copy the files in other locations but they always end up in temporary folders and are never found during the build

Can you help me sort this out?

samcmill · 2021-02-21T20:07:14Z

What is your Singularity build command line?

VI-gha · 2021-02-21T21:37:22Z

sudo singularity build aocc.sif aocc.def INFO: Starting build...Getting image source signaturesCopying blob d519e2592276 skipped: already exists Copying blob d22d2dfcfa9c skipped: already exists Copying blob b3afe92c540b [--------------------------------------] 0.0b / 0.0bCopying config d0de749f24 done Writing manifest to image destinationStoring signatures2021/02/21 22:28:50 info unpack layer: sha256:d519e2592276828ca171d85e0532899cd4f98c70f5c697b45fa2e126e9f9fe492021/02/21 22:28:51 info unpack layer: sha256:d22d2dfcfa9cd230ed3c47defec2670d45081598c721dd85cafc34ea459f970e2021/02/21 22:28:51 info unpack layer: sha256:b3afe92c540b778c64ca316d1e679d55d2d2e812e450f516a808ee591f0c3f77INFO: Copying aocc-compiler-2.3.0.tar to /tmp/build-temp-807170343/rootfs/var/tmp/aocc-compiler-2.3.0.tar(...)+ tar -xvf /var/tmp/aocc-compiler-2.3.0.tar -C /usr/localtar: /var/tmp/aocc-compiler-2.3.0.tar: Cannot open: No such file or directorytar: Error is not recoverable: exiting nowFATAL: While performing build: while running engine: exit status 2 Files copies inside singularity end up in a temporary folder (here it is /tmp/build-temp-807170343/rootfs/var/tmp instead of /var/tmp) and cannot be used during the build On Sunday, 21 February 2021, 21:07:34 CET, Scott McMillan <notifications@github.com> wrote: What is your Singularity build command line? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

VI-gha · 2021-02-22T08:51:29Z

I have tried something similar with the AOCC compiler (which is much smaller) and get the same type of error
This reproduces it (I just kept the bare minimum)

BootStrap: docker
From: ubuntu:18.04

%files
aocc-compiler-2.3.0.tar /var/tmp

%post
tar -xvf /var/tmp/aocc-compiler-2.3.0.tar -C /usr/local
cd /usr/local/aocc-compiler-2.3.0
bash install.sh

What is fundamentally wrong with that?

samcmill · 2021-02-22T14:52:46Z

What version of Singularity and OS are you using? I can't reproduce the issue with 3.5.3 on Linux (CentOS 7).

It might be apptainer/singularity#5514? That only impacts 3.6 and later on WSL. The workaround in that case is to use a directory that is not bind mounted from the host into the container, so please try hpccm --working-directory /other/path ...

VI-gha · 2021-02-22T15:57:09Z

I am using singularity version 3.6.4 and for the os NAME="Ubuntu"VERSION="20.04.2 LTS (Focal Fossa)" On Monday, 22 February 2021, 15:53:09 CET, Scott McMillan <notifications@github.com> wrote: What version of Singularity and OS are you using? I can't reproduce the issue with 3.5.3 on Linux (CentOS 7). It might be apptainer/singularity#5514? That only impacts 3.6 and later on WSL. The workaround in that case is to use a directory that is not bind mounted from the host into the container, so please try hpccm --working-directory /other/path ... — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

VI-gha · 2021-02-22T16:27:20Z

I am not sure I understand what you mean by "hpccm --working-directory /other/path ..." I tried this hpccm recipe: # Working directoryStage0 += workdir(directory='/opt/src') # AOCCStage0 += copy(src='aocc-compiler-2.3.0.tar', dest='/opt/src/aocc-compiler-2.3.0.tar')Stage0 += shell(commands=['tar -xvf /opt/src/aocc-compiler-2.3.0.tar -C /opt/src', 'cd /opt/src/aocc-compiler-2.3.0', 'bash install.sh']) which translated into: BootStrap: dockerFrom: ubuntu:18.04%post . /.singularity.d/env/10-docker*.sh %post cd / mkdir -p /opt/src cd /opt/src %files aocc-compiler-2.3.0.tar /opt/src/aocc-compiler-2.3.0.tar %post cd / tar -xvf /opt/src/aocc-compiler-2.3.0.tar -C /opt/src cd /opt/src/aocc-compiler-2.3.0 bash install.sh and still gives the same error as before INFO: Starting build...Getting image source signaturesCopying blob d519e2592276 skipped: already exists Copying blob d22d2dfcfa9c skipped: already exists Copying blob b3afe92c540b [--------------------------------------] 0.0b / 0.0bCopying config d0de749f24 done Writing manifest to image destinationStoring signatures2021/02/22 17:20:34 info unpack layer: sha256:d519e2592276828ca171d85e0532899cd4f98c70f5c697b45fa2e126e9f9fe492021/02/22 17:20:35 info unpack layer: sha256:d22d2dfcfa9cd230ed3c47defec2670d45081598c721dd85cafc34ea459f970e2021/02/22 17:20:35 info unpack layer: sha256:b3afe92c540b778c64ca316d1e679d55d2d2e812e450f516a808ee591f0c3f77INFO: Copying aocc-compiler-2.3.0.tar to /tmp/build-temp-966361227/rootfs/opt/src/aocc-compiler-2.3.0.tarINFO: Running post scriptlet+ . /.singularity.d/env/10-docker2singularity.sh+ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin+ cd /+ mkdir -p /opt/src+ cd /opt/src+ cd /+ tar -xvf /opt/src/aocc-compiler-2.3.0.tar -C /opt/srctar (child): xz: Cannot exec: No such file or directorytar (child): Error is not recoverable: exiting nowtar: Child returned status 2tar: Error is not recoverable: exiting nowFATAL: While performing build: while running engine: exit status 2 This is exactly what happens with intel_psxe On Monday, 22 February 2021, 16:55:42 CET, vru.inbri@yahoo.co.uk <vru.inbri@yahoo.co.uk> wrote: I am using singularity version 3.6.4 and for the os NAME="Ubuntu"VERSION="20.04.2 LTS (Focal Fossa)" On Monday, 22 February 2021, 15:53:09 CET, Scott McMillan <notifications@github.com> wrote: What version of Singularity and OS are you using? I can't reproduce the issue with 3.5.3 on Linux (CentOS 7). It might be apptainer/singularity#5514? That only impacts 3.6 and later on WSL. The workaround in that case is to use a directory that is not bind mounted from the host into the container, so please try hpccm --working-directory /other/path ... — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

samcmill · 2021-02-23T15:20:29Z

The --working-directory option changes the location inside the container to use as the working (staging) directory.

For example, if the recipe is:

Stage0 += baseimage(image='ubuntu:18.04')

Stage0 += intel_psxe(eula=True, license='license.lic', tarball='parallel_studio_xe_2020_update4_cluster_edition.tgz')

Then hpccm --recipe 341.py --format singularity --working-directory /opt outputs:

BootStrap: docker
From: ubuntu:18.04
%post
    . /.singularity.d/env/10-docker*.sh

# Intel Parallel Studio XE
%post
    apt-get update -y
    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
        build-essential \
        cpio
    rm -rf /var/lib/apt/lists/*
%files
    parallel_studio_xe_2020_update4_cluster_edition.tgz /opt/parallel_studio_xe_2020_update4_cluster_edition.tgz
%files
    license.lic /opt/license.lic
%post
    cd /
    mkdir -p /opt && tar -x -f /opt/parallel_studio_xe_2020_update4_cluster_edition.tgz -C /opt -z
    sed -i -e 's/^#\?\(COMPONENTS\)=.*/\1=DEFAULTS/g' \
        -e 's|^#\?\(PSET_INSTALL_DIR\)=.*|\1=/opt/intel|g' \
        -e 's/^#\?\(ACCEPT_EULA\)=.*/\1=accept/g' \
        -e 's/^#\?\(ACTIVATION_TYPE\)=.*/\1=license_file/g' \
        -e 's|^#\?\(ACTIVATION_LICENSE_FILE\)=.*|\1=/opt/license.lic|g' /opt/parallel_studio_xe_2020_update4_cluster_edition/silent.cfg
    cd /opt/parallel_studio_xe_2020_update4_cluster_edition && ./install.sh --silent=silent.cfg
    rm -rf /opt/parallel_studio_xe_2020_update4_cluster_edition.tgz /opt/parallel_studio_xe_2020_update4_cluster_edition
%post
    cd /
    echo "source /opt/intel/compilers_and_libraries/linux/bin/compilervars.sh intel64" >> /etc/bash.bashrc

Note that the tarball is copied to /opt rather than /var/tmp.

This looks like a Singularity issue, not a HPCCM issue? Have you filed an issue at https://github.com/hpcng/singularity/issues?

VI-gha · 2021-02-23T21:11:00Z

Thanks, I did not know this option with --working-directory /opt I am going to give it a try On Tuesday, 23 February 2021, 16:20:56 CET, Scott McMillan <notifications@github.com> wrote: The --working-directory option changes the location inside the container to use as the working (staging) directory. For example, if the recipe is: Stage0 += baseimage(image='ubuntu:18.04') Stage0 += intel_psxe(eula=True, license='license.lic', tarball='parallel_studio_xe_2020_update4_cluster_edition.tgz') Then hpccm --recipe 341.py --format singularity --working-directory /opt outputs: BootStrap: docker From: ubuntu:18.04 %post . /.singularity.d/env/10-docker*.sh # Intel Parallel Studio XE %post apt-get update -y DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ build-essential \ cpio rm -rf /var/lib/apt/lists/* %files parallel_studio_xe_2020_update4_cluster_edition.tgz /opt/parallel_studio_xe_2020_update4_cluster_edition.tgz %files license.lic /opt/license.lic %post cd / mkdir -p /opt && tar -x -f /opt/parallel_studio_xe_2020_update4_cluster_edition.tgz -C /opt -z sed -i -e 's/^#\?\(COMPONENTS\)=.*/\1=DEFAULTS/g' \ -e 's|^#\?\(PSET_INSTALL_DIR\)=.*|\1=/opt/intel|g' \ -e 's/^#\?\(ACCEPT_EULA\)=.*/\1=accept/g' \ -e 's/^#\?\(ACTIVATION_TYPE\)=.*/\1=license_file/g' \ -e 's|^#\?\(ACTIVATION_LICENSE_FILE\)=.*|\1=/opt/license.lic|g' /opt/parallel_studio_xe_2020_update4_cluster_edition/silent.cfg cd /opt/parallel_studio_xe_2020_update4_cluster_edition && ./install.sh --silent=silent.cfg rm -rf /opt/parallel_studio_xe_2020_update4_cluster_edition.tgz /opt/parallel_studio_xe_2020_update4_cluster_edition %post cd / echo "source /opt/intel/compilers_and_libraries/linux/bin/compilervars.sh intel64" >> /etc/bash.bashrc Note that the tarball is copied to /opt rather than /var/tmp. This looks like a Singularity issue, not a HPCCM issue? Have you filed an issue at https://github.com/hpcng/singularity/issues? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

VI-gha · 2021-02-25T09:42:56Z

@samcmill

Thanks again Scott, it worked when using --working-directory /opt
I guess that others will have the same problem: would it not be better if that was documented or if the intel_psxe building block was modified to use %setup+cp instead of %files?

%setup
    cp parallel_studio_xe_2020_update4_cluster_edition.tgz /var/tmp/parallel_studio_xe_2020_update4_cluster_edition.tgz
    cp license.lic /var/tmp/license.lic

samcmill · 2021-02-25T16:52:36Z

Thanks @VI-gha. I need to investigate the apparent Singularity changed %files behavior. The solution may be to use %setup, I'm not sure.

I'm going to close this issue and open a new one focused on the Singularity behavior change.

samcmill closed this as completed Feb 25, 2021

samcmill mentioned this issue Feb 25, 2021

Singularity %files behavior change in 3.6 and later #345

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Files copied inside singularity not found (intel_psxe building block) #341

Files copied inside singularity not found (intel_psxe building block) #341

VI-gha commented Feb 21, 2021

samcmill commented Feb 21, 2021

VI-gha commented Feb 21, 2021 via email

VI-gha commented Feb 22, 2021

samcmill commented Feb 22, 2021

VI-gha commented Feb 22, 2021 via email

VI-gha commented Feb 22, 2021 via email

samcmill commented Feb 23, 2021

VI-gha commented Feb 23, 2021 via email

VI-gha commented Feb 25, 2021

samcmill commented Feb 25, 2021

Files copied inside singularity not found (intel_psxe building block) #341

Files copied inside singularity not found (intel_psxe building block) #341

Comments

VI-gha commented Feb 21, 2021

base image

Intel PSXE

samcmill commented Feb 21, 2021

VI-gha commented Feb 21, 2021 via email

VI-gha commented Feb 22, 2021

samcmill commented Feb 22, 2021

VI-gha commented Feb 22, 2021 via email

VI-gha commented Feb 22, 2021 via email

samcmill commented Feb 23, 2021

VI-gha commented Feb 23, 2021 via email

VI-gha commented Feb 25, 2021

samcmill commented Feb 25, 2021