Add bitwise comparison step for FLUXNET test suite

[SeanBryan51]

This change adds the ability for benchcab to run bitwise comparisons between NetCDF output files using the nccmp command. Comparisons are made between outputs that differ in their realisation and are matching in all other configurations (science configurations and meteorological forcing). Write standard output from comparison tasks on failure to the runs/site/analysis/bitwise-comparisons directory.

Since multiple realisations can be specified, comparisons are made between all pair wise combinations of realisations.

This change removes the --no-submit optional argument from benchcab fluxnet-run-tasks and instead submit a PBS job only when running benchcab run or benchcab fluxnet. We do this so that we can run the bitwise comparison step in the same job script used to run CABLE.

The comparison step can be run in isolation by executing benchcab fluxnet-bitwise-cmp. However, this should ideally be executed on a compute node (inside a PBS job for example).

This change also refactors the parallelisation scheme used for running CABLE tasks and comparison tasks so that workers fetch tasks from a multiprocessing.Queue object. Previously, if a process had completed the CABLE tasks it was allocated, it will remain idle until all other processes had completed their allocated tasks. This change prevents processes from idling if tasks are still yet to be completed.

The comparison step can be skipped by specifying --skip fluxnet-bitwise-cmp to composite commands such as benchcab run and benchcab fluxnet.

Fixes #32
Fixes #77

[SeanBryan51]

TODO

• measure the time it takes to run benchcab fluxnet-run-bitwise-cmp in a PBS job for five site and forty two site case with default science configurations
• make benchcab fluxnet-run-bitwise-cmp an automated step when runnning benchcab run. Submit in single job script?
• use diffn instead of diff
• run-bitwise-cmp -> bitwise-cmp
• write standard output from cdo diff to file on failure

[SeanBryan51]

Now using nccmp instead of cdo diffn. This is because cdo diffn was using an unreasonable amount of memory per comparison and was causing processes to be killed. The memory usage of cdo diffn linearly increases with respect to time and may be a memory leak in the tool.

Tested for cdo versions 1.7.2, 1.9.8, 1.9.10 and 2.0.5

[ccarouge]

It looks great. Just one comment to check for patch to run the comparison.

[ccarouge]

Good idea to move the no-submit option to the fluxnet command.

[ccarouge]

Should we add a test on branch_patch so we don't run the test if only one of the tasks has a patch applied? Since in this case, we are fairly certain the science configurations are different.

If both branches have a patch, we should keep the comparison as the patches could be identical.

[SeanBryan51]

Yep I agree with those cases where a user does not need to run a regression test. However, I think the best approach would be for the user manually switch off the regression test (we have an issue for this: #77).

It might be counter productive trying to guess when the user wants to switch on/off the regression test. For example, we might want to do a regression test as a sanity check.