Buffer size for sort

[pappewaio]

Great, it seems we agree on most stuff here.

5. Btw, is there a way to control how much memory the sort is allowed to use? Or by default it it'll use whatever is available?

When cleaning, sort doesn't need to use much memory, and it will use default --buffer-size, which is calculated on the fly. For many sorts in the cleaning, adding --buffer-size, won't have an effect as sort is used with a pipe "|", and only few MBs are being used. So far no one has experienced any problems with the default settings. It might be worth adding parallelisation and --buffer-size as options as well though. It will likely make things run much faster, provided more cpus can be accessed.

I have added sort --buffer-size=20G for the sorting during snpdb specific preparation, and I think it is because adding parallelisation failed the default buffer calculation. It should be added as an option in nextflow.config, and the behaviour should be explained somewhere.

--buffer-size was explained well in this forum post:
https://stackoverflow.com/questions/37514283/gnu-sort-default-buffer-size

Originally posted by @pappewaio in #256 (comment)