Use separate script for long stage/unstage commands

[bentsherman]

Closes #2852

TODO:

• e2e test

[bentsherman]

CI tests were double triggered for push and pull_request, @ewels did we do something wrong?

[ewels]

It's because your fork is out of date. I think if you hit "Update branch" it should work as expected... 🤞🏻

[ewels]

Sorry, couldn't resist hitting the button to find out 😆

Looking good now though!

[bentsherman]

Looking at #2118 , I believe this check was added as a partial solution:

nextflow/modules/nextflow/src/main/groovy/nextflow/executor/SimpleFileCopyStrategy.groovy

Lines 132 to 133 in 6ef14c0

	if( len<100 )
	delete << "rm -f ${Escape.path(stageName)}"

So can we safely remove this check? Now it will be in the .command.stage script anyway.

[pditommaso]

Made some improvements:

• write files using newBufferedWriter to avoid allocation a new byte[] buffer that for large file can be expensive
• use bash instead of source to read the stage command
• use NXF_WRAPPER_STAGE_FILE_THRESHOLD as bytes threshold instead of num of files to better control it

It should be added an integration tests in tests/ folder before merging it

[pditommaso]

Considering the Slurm limit is 5MB (see here), i've creased the default to 1 MB

[pditommaso]

Let's merge this. Solve #2852

[ewels]