Re-work upload clarification from #5206.

[jmchilton]

Re-work upload clarification from #5206.

See post-merge discussion on that issue.

[mvdbeek]

👍 I think that's correct now.

[nsoranzo]

This looks good to me, but do we have any test for the purge_source upload option? It seems that should have failed before.

[jmchilton]

@nsoranzo Well... in fact... https://github.com/galaxyproject/galaxy/blob/dev/test/integration/test_upload_configuration_options.py#L264.

        # Purge is set by default so this should be gone.
        # ... but it isn't - is this a bug? Are only certain kinds of uploads purged?
        # assert not os.path.exists(ftp_path)

[jmchilton]

Well I played a bit more with that test case and it seems like purge would only ever be used with non-binary data if the to posix lines option is disabled - because usually Galaxy will create a new file and not purge the original. I may open a PR to clarify some of this in testing. I guess an open question is if we should purge those files that we don't move - just to keep things clean and consistent across datatypes and data content.

[nsoranzo]

@jmchilton @mvdbeek I just find out (after a long session of git bisect) that this is the commit causing the BioBlend test failures at https://travis-ci.org/galaxyproject/bioblend/builds/339232363 for 18.01 and dev. When testing upload to libraries from filesystem paths, the files to upload are moved (instead of copied) at https://github.com/galaxyproject/bioblend/blob/master/bioblend/_tests/TestGalaxyLibraries.py#L61 , so the upload of the same files in the next line fails.

I think we should modify tools/data_source/upload.py as follows:

purge_source = dataset.get('purge_source', True) and not run_as_real_user and dataset.type not in ('server_dir', 'path_paste')