mcpltool: Possibility to force merge or ignore comments/blobs when comparing headers

[willend]

Juggling mcpl output files split using the DG framework ess_mcplextra_filterfile, I wanted to merge two mcpl files, but mcpltool complains that

 mcpltool --merge full /scratch/MCNP/BIFROST/with_SM_at_sample.mcpl.gz.mcpl.gz /scratch/MCNP/BIFROST/with_SM_at_2m.mcpl.gz.mcpl.gz 
ERROR: Requested files are incompatible for merge as they have different header info.

Run with -h or --help for usage information

(Admittedly this is kind of a stupid example since they originated from the same MCNP ssw write in the first place...)

Isn't mcpltool a little picky here since the only difference is stuff in the comments? Headers of the two files are visible below:

pkwi@elearn1:~/MCPL-filter$ mcpltool  /scratch/MCNP/BIFROST/with_SM_at_sample.mcpl.gz.mcpl.gz
Opened MCPL file with_SM_at_sample.mcpl.gz.mcpl.gz:

  Basic info
    Format             : MCPL-3
    No. of particles   : 1307
    Header storage     : 286 bytes
    Data storage       : 47052 bytes

  Custom meta data
    Source             : "MCNP6"
    Number of comments : 3
          -> comment 0 : "SSW file from MCNP6 converted with ssw2mcpl (from MCPL release v1.2.3)"
          -> comment 1 : "SSW metadata: [kods='mcnp', vers='6.mpi', title='Input File:']"
          -> comment 2 : "File edited by ess_mcplextra_filterfile (filter='sqrt(x^2+y^2)>250cm', norig=4331257)"
    Number of blobs    : 0

  Particle data format
    User flags         : no
    Polarisation info  : no
    Fixed part. type   : no
    Fixed part. weight : no
    FP precision       : single
    Endianness         : little
    Storage            : 36 bytes/particle

index     pdgcode   ekin[MeV]       x[cm]       y[cm]       z[cm]          ux          uy          uz    time[ms]      weight
    0        2112  0.00069022       11652       10751      12.879     0.73488      0.6782 -0.00018831     0.43607  9.3117e-21
    1        2112   6.492e-05       11652       10750       14.12     0.73493     0.67814  2.7892e-05       1.423  1.6457e-17
    2        2112  8.6716e-06       11652       10751       12.88     0.73494     0.67813 -0.00013716      3.8942  4.1574e-21
    3        2112  5.1535e-07       11653       10749      13.946     0.73503     0.67803  4.3025e-05      15.964  1.5176e-24
    4        2112  1.7476e-07       11652       10751      14.251     0.73486     0.67822 -7.6372e-05       27.64  2.6269e-32
    5        2112  1.0504e-05       11653       10750      13.983     0.73488      0.6782 -0.00013821      3.5348  9.9005e-35
    6        2112   9.009e-08       11653       10750      15.695     0.73489     0.67819  1.4811e-06      38.219  2.6135e-38
    7        2112  7.8446e-08       11654       10749      13.981     0.73493     0.67815 -8.6636e-05      41.015  1.6222e-38
    8        2112  7.2098e-08       11652       10750      13.146     0.73483     0.67825 -0.00011361      42.798  7.0125e-39
    9        2112      27.396       11654       10748      18.655     0.73495     0.67812  3.6056e-05   0.0022445  2.0645e-37
pkwi@elearn1:~/MCPL-filter$ mcpltool /scratch/MCNP/BIFROST/with_SM_at_2m.mcpl.gz.mcpl.gz             
Opened MCPL file with_SM_at_2m.mcpl.gz.mcpl.gz:

  Basic info
    Format             : MCPL-3
    No. of particles   : 4329950
    Header storage     : 286 bytes
    Data storage       : 155878200 bytes

  Custom meta data
    Source             : "MCNP6"
    Number of comments : 3
          -> comment 0 : "SSW file from MCNP6 converted with ssw2mcpl (from MCPL release v1.2.3)"
          -> comment 1 : "SSW metadata: [kods='mcnp', vers='6.mpi', title='Input File:']"
          -> comment 2 : "File edited by ess_mcplextra_filterfile (filter='sqrt(x^2+y^2)<250cm', norig=4331257)"
    Number of blobs    : 0

  Particle data format
    User flags         : no
    Polarisation info  : no
    Fixed part. type   : no
    Fixed part. weight : no
    FP precision       : single
    Endianness         : little
    Storage            : 36 bytes/particle

index     pdgcode   ekin[MeV]       x[cm]       y[cm]       z[cm]          ux          uy          uz    time[ms]      weight
    0        2112  3.9421e-08       148.1      134.41      14.467     0.80934      0.5873   0.0073535     0.99758  5.2378e-10
    1        2112  3.9421e-08       146.4      136.26      15.093     0.80146     0.59792    0.012251     0.99495  4.3356e-10
    2        2112  3.9421e-08      147.91      134.63       13.05     0.81118      0.5848 -0.00023405     0.99508  1.1035e-10
    3        2112  3.9421e-08      146.86      135.77      17.678     0.78111     0.62413    0.018017     0.98616  2.8613e-05
    4        2112   4.347e-08      145.88      136.82      14.361     0.77645     0.63018  -0.0011231     0.95463  4.3021e-06
    5        2112   4.889e-08      145.25      137.49      15.559     0.77294     0.63446   0.0056494     0.91891  1.2184e-05
    6        2112  4.6901e-08      150.67      131.53      14.201     0.80003     0.59996  0.00040006     0.93111  6.1158e-06
    7        2112  4.3054e-08      149.77      132.55      18.643     0.79622      0.6045    0.024833     0.95529  7.0589e-06
    8        2112  6.9368e-08      148.67      133.78      12.786     0.79096      0.6118  -0.0091003     0.82267  5.3271e-06
    9        2112  6.0126e-08      147.38       135.2       12.55      0.7846     0.61992   -0.010371     0.85841  2.2517e-06
pkwi@elearn1:~/MCPL-filter$ 

[tkittel]

Admittedly it is being very conservative and picky, in that it wants to make sure that the merged file has consistent comments as well.

But the use-case of forcing the merge to take place anyway (possibly with comments from both files kept in a clearly denoted fashion) is something that has come up many times. So I definitely support some sort of development in this direction.

Just need to find the time :-)

[willend]

Sounds good - did you find the time yet? ;-) (Just kidding)

As I more or less needed the possibility "now" I have hacked up yet another little crazy fucker of a McStas instrument file - MCPL_merge.instr: https://github.com/McStasMcXtrace/McCode/blob/master/mcstas-comps/examples/MCPL_merge.instr

(- needs an updated MCPL_input to work: https://github.com/McStasMcXtrace/McCode/blob/master/mcstas-comps/misc/MCPL_input.comp)

[willend]

• Have a nice weekend also of course :-)

[tkittel]

But what if I have 11 mcpl files? ;-)

Just kidding, good that you have a short term workaround. I guess it should not be super difficult to simply add a "--merge-and-strip-comments" feature to mcpltool which does something similar and just nukes all comments (but inserts a new warning comment). Care must of course still be taken to enable the right settings (double precision/polarisation/userflags) to ensure that no particle is lost.

[tkittel]

And I am to late to return the wishes for a nice weekend, so please have a nice next weekend instead ;-)

[tkittel]

In MCPL v1.3.0 which was just released, the mcpltool has a new --forcemerge option as requested:

  --forcemerge [--keepuserflags] FILEOUT FILE1 FILE2 ... FILEN
               Like --merge but works with incompatible files as well, at the
               heavy price of discarding most metadata like comments and blobs.
               Userflags will be discarded unless --keepuserflags is specified.

It will select the format of the output files so as to be able to accommodate all particles in the input. Example: 15 input files are use single-precision storage and 1 use double-precision storage, so the output will use double-precision storage.
Another example: If any input file has polarisation enabled, the output will have it enabled as well.

It performs all particle transfers loss-lessly (i.e. nothing lost by unpacking+repacking) via the new mcpl_transfer_last_read_particle function from the C API.

It is documented on https://mctools.github.io/mcpl/usage_cmdline/

And just to be clear: It is strongly recommended to use the standard --merge whenever possible! The new --forcemerge option is intended as a last-resort only.