adding support for --sparse when creating ext3

[vsoch]

Description of the Pull Request (PR):

this will close issue #610. We basically want to add support for a sparse overlay, and we can do that by using truncate -s (size) instead of dd. In the case the command is not found, an error is issued and the user can decide to install or not use it. I think it would be overkill to hard code information about all OS versions / support but if anyone has ideas for how to reasonably do this we can return a more specific error message. I am not running the e2e tests locally but will see how they do in the CI! :)

Signed-off-by: vsoch vsoch@users.noreply.github.com

This fixes or addresses the following GitHub issues:

• Fixes singularity overlay create: support sparse ext3 images #610

Question: are we still adding changes to the changelog? It looks like they are added after release (at least the top version is already released and there isn't space for develop changes above that)

[dtrudg]

General thoughts at this stage...

• Does it make sense to allow someone to use --sparse when adding an overlay to a SIF file? The code currently allows this I believe. If it makes sense, and is permitted, then it should be tested also.

[vsoch]

Looks like my one test failed - is truncate installed in the testing environment, and is there a way to see output for why it failed?

        --- FAIL: TestE2E/PAR/OVERLAY (0.00s)
            --- FAIL: TestE2E/PAR/OVERLAY/create (27.67s)
                --- PASS: TestE2E/PAR/OVERLAY/create/create_ext3_overlay_with_small_size (0.29s)
                --- PASS: TestE2E/PAR/OVERLAY/create/create_ext3_sparse_overlay_image (0.64s)
                --- FAIL: TestE2E/PAR/OVERLAY/create/create_ext3_overlay_image (0.23s)
                --- PASS: TestE2E/PAR/OVERLAY/create/check_ext3_overlay_size (0.53s)
                --- PASS: TestE2E/PAR/OVERLAY/create/create_ext3_overlay_with_an_existing_image (0.15s)
                --- PASS: TestE2E/PAR/OVERLAY/create/create_ext3_overlay_with_dir (0.92s)
                --- PASS: TestE2E/PAR/OVERLAY/create/check_overlay_dir_permissions (0.54s)
                --- PASS: TestE2E/PAR/OVERLAY/create/create_ext3_overlay_image_in_unsigned_SIF (0.96s)
                --- PASS: TestE2E/PAR/OVERLAY/create/create_ext3_overlay_image_in_SIF_with_an_existing_overlay (0.13s)
                --- PASS: TestE2E/PAR/OVERLAY/create/create_ext3_overlay_image_in_signed_SIF (0.20s)
                --- PASS: TestE2E/PAR/OVERLAY/create/create_ext3_overlay_image_in_encrypted_SIF (0.20s)

[dtrudg]

Looks like my one test failed - is truncate installed in the testing environment, and is there a way to see output for why it failed?

Yes - take the name of the failing test create_ext3_overlay_image, and search for it in the e2e output. All test output is present, but because of parallelization the output of tests is interleaved, so you have to search for it. You need to be looking in the full output (the blue "Download full output as a file") on the CircleCI page.

For this test, the failure is:

=== CONT  TestE2E/PAR/OVERLAY/create/create_ext3_overlay_image
    overlay.go:178: 
        "/usr/local/bin/singularity overlay create --size 128 /tmp/stest.216691604/overlay3990262076/image.ext3" output:
        FATAL:   EXT3 overlay image /tmp/stest.216691604/overlay3990262076/image.ext3 already exists
        
    overlay.go:178: got 255 as exit code and was expecting 0: exit status 255
        waiting for command "/usr/local/bin/singularity overlay create --size 128 /tmp/stest.216691604/overlay3990262076/image.ext3"
        github.com/sylabs/singularity/e2e/internal/e2e.TestEnv.RunSingularity.func1
        	github.com/sylabs/singularity/e2e/internal/e2e/singularitycmd.go:664
        testing.tRunner
        	testing/testing.go:1439
        runtime.goexit
        	runtime/asm_amd64.s:1571

Looks like the filename has already been used by a preceding test, that didn't delete that file yet. Either the earlier test needs to remove the file, or (safer), you can use a different filename here.

[vsoch]

woo green @dtrudg !

[dtrudg]

I did realize we need CHANGELOG.md entries for this, and your instance stats work - but I'll add those in a separate PR, so this can be merged with the other things pending.

Thanks @vsoch !

[vsoch]

woohoo! Thanks @dtrudg!