Improve file type detection of NetCDF and HDF5

[pdurbin]

According to Wikipedia NetCDF and HDF5 have magic numbers that should let us detect these file types more easily and reliably than guessing based on file extensions.

NetCDF magic number

CDF\001
\211HDF\r\n\032\n

HDF5 magic number

\211HDF\r\n\032\n

I brought this up at standup today and here are some notes from the discussion:

• We should see if JHOVE can detect them.
• Normally, detecting file types by seeking into files is part of detecting tabular files, specifically.
• Given that NetCDF can be big, this might be a case where switching to a ranged request to find the signature might be important.

We should add some NetCDF and HDF5 files to https://github.com/IQSS/dataverse-sample-data to test with, at some point.

Related:

• HDF5 files show type "unknown" dataverse.harvard.edu#23

[mreekie]

Updated Project status's

[mreekie]

@pdurbin dumb question.

• This issue was linked to PR 2 weeks ago.
  • per the process it was then removed from the dataverse sprint board.
• So when the PR makes it through QA this issue will be automatically closed right?

[pdurbin]

Yes, the issue will be closed when it is merged.

In our PRs we use the magic "closes" syntax.

We'll write something like this:

"Closes #12345"

This makes an association between the PR and the issue (or multiple issues). When you merge the PR, the issue (or multiple issues) is closed.

Please see https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/using-keywords-in-issues-and-pull-requests#linking-a-pull-request-to-an-issue