separate out non-compliant data files

[prjemian]

some of the data files here are not compliant with the NeXus standard

there could be several categories:

• compliant HDF4 files (deprecated for new use)
• compliant XML files (deprecated for new use)
• compliant HDF5 files
• non-compliant data files
• non-compliant data files to be kept for testing

[prjemian]

related to #2

[prjemian]

2020 Code Camp: sounds like a good code camp activity - run a validator against all the data files and categorize

Can jenkins be configured to validate on commit or periodically? Might need to run periodically due to changes in NXDL files in the definitions repository.

[prjemian]

There is a file critique.py, added in 2015, that performs a simple assessment of whether a given file is a NeXus HDF5 file, an HDF5 file, or none of these.

[prjemian]

Using the revision of critique.py from the branch:

(py27) mintadmin@mint-vm:~/.../NeXus/exampledata$ ./critique.py 

.
+
:.gitignore:  not HDF5 file
:.project:  not HDF5 file
:.pydevproject:  not HDF5 file
:MANIFEST.in:  not HDF5 file
:README.rst:  not HDF5 file
:critique.py:  not HDF5 file
:simple3D.h5:  HDF5 file
:verysimple.xml:  not HDF5 file
:writer_1_3.h5:  HDF5 file
:writer_1_3__niac2014.h5:  NeXus HDF5 file

./.vscode
+++++++++
:launch.json:  not HDF5 file
:settings.json:  not HDF5 file

./ANSTO
+++++++
:PLP0006018.nx.hdf:  HDF5 file

./APS/CCDImageServer
++++++++++++++++++++
:README.txt:  not HDF5 file
:Smither400.3_apr0300051.hdf:  not HDF5 file
:Smither400.3_apr0300052.hdf:  not HDF5 file
:Smither400.3_apr0300053.hdf:  not HDF5 file
:Smither400.3_apr0300054.hdf:  not HDF5 file
:Smither400.3_apr0300055.hdf:  not HDF5 file
:recon_0123.hdf:  not HDF5 file
:recon_0124.hdf:  not HDF5 file
:recon_0125.hdf:  not HDF5 file

./APS/EPICSareaDetector/NeXus-plugin
++++++++++++++++++++++++++++++++++++
:AgBehenate_228.hdf5:  HDF5 file

./APS/NXsas
+++++++++++
:create_nexus_data.py:  not HDF5 file
:nexus-example.hdf5:  NeXus HDF5 file
:xture.txt:  not HDF5 file

./APS/other
+++++++++++
:ID34_not_complete.h5:  HDF5 file

./APS/scan2nexus
++++++++++++++++
:14BMC_0015.mda:  not HDF5 file
:14BMC_0015.nexus:  not HDF5 file
:14BMC_0015.xml:  not HDF5 file
:2iddf_0106.mda:  not HDF5 file
:2iddf_0106.nexus:  not HDF5 file
:2iddf_0106.xml:  not HDF5 file
:README:  not HDF5 file
:mts_0347.mda:  not HDF5 file
:mts_0347.nexus:  not HDF5 file
:mts_0347.xml:  not HDF5 file
:mts_0348.mda:  not HDF5 file
:mts_0348.nexus:  not HDF5 file
:mts_0348.xml:  not HDF5 file
:sample.mda.text:  not HDF5 file
:sample1.mda:  not HDF5 file
:sample1.nexus:  not HDF5 file
:sample1.xml:  not HDF5 file

./APS/tomo
++++++++++
:Tomography_metadata.hdf:  not HDF5 file
:Tomography_raw.hdf:  not HDF5 file

./DLS
+++++
:README.rst:  not HDF5 file

./DLS/NXquadric
+++++++++++++++
:sample_capillary.nxs:  HDF5 file

./DLS/i03_i04_NXmx
++++++++++++++++++
:README.txt:  not HDF5 file
:Therm_6_2.nxs:  NeXus HDF5 file

./DLS/i16
+++++++++
:538039.nxs:  HDF5 file

./DLS/i16/538039-pilatus100k-files
++++++++++++++++++++++++++++++++++
:538039.hdf:  HDF5 file

./DLS/p45
+++++++++
:p45-1168-mic.hdf5:  HDF5 file
:p45-1168.nxs:  HDF5 file
:p45-2194.nxs:  HDF5 file
:p45-316.nxs:  HDF5 file

./DLS/reflections
+++++++++++++++++
:thaumatin_integrated.nxs:  HDF5 file

./IPNS/LRMECS
+++++++++++++
:lrcs3701.nx5:  HDF5 file
:lrcs3701.nxs:  not HDF5 file

./Soleil
++++++++
:file_1.nxs:  HDF5 file
:file_2.nxs:  HDF5 file

./SwissFEL
++++++++++
:README:  not HDF5 file
:lyso009a_0087.JF07T32V01_master.h5:  HDF5 file

./code/hdf4
+++++++++++
:NXtest.hdf:  not HDF5 file
:dmc01.hdf:  not HDF5 file
:dmc02.hdf:  not HDF5 file

./code/hdf5
+++++++++++
:NXtest.h5:  HDF5 file
:dmc01.h5:  HDF5 file
:dmc02.h5:  HDF5 file
:focus2007n001335.hdf:  HDF5 file
:sans2009n012333.hdf:  HDF5 file

./code/xml
++++++++++
:NXtest.xml.txt:  not HDF5 file
:dmc01.xml.txt:  not HDF5 file
:dmc02.xml.txt:  not HDF5 file

./nxpdb
+++++++
:4n8z.cif:  not HDF5 file
:4n8z.h5:  HDF5 file
:4n8z.h5.cif:  not HDF5 file
:README.rst:  not HDF5 file

[prjemian]

That output is reST and the code is python2. OK for now. Will this help separate out the non-compliant data files?

[PeterC-DLS]

That output is reST and the code is python2. OK for now. Will this help separate out the non-compliant data files?

Yup, I will make subdirectories to label hdf5, hdf4 and other content.