DM-14197: Make obs_test data ingestible

[kfindeisen]

This PR makes it possible to ingest obs_test files into an external repository, in order to test code that uses ingestion. Rather than modify the files to be ingestible with default settings, I've added a custom ParseTask and some config override files.

This PR also updates obs_test to the modern copyright format (so that I could use it when adding obs/test/ingestCalibs.py) and corrects some formatting and reproducibility issues in the data readme.

[r-owen]

If you added a few headers to the existing files could you get rid of TestCalibsParseTask and some of these overrides? If so, I recommend doing that in order to simplify the approach and make the data more realistic and a better fit to standard ingestion. That said, this approach works and is acceptable.

[r-owen]

Consider return date.split("T")[0]. Like your code it will return everything up to, but not including, the first T. If there is no T then it returns the whole date.

Using split in this way is a common Python idiom.

[r-owen]

Consider:

_knownTypes = frozenset(("bias", "flat", "defect"))  # known file types as a class variable
def getCalibType(...):
    ....
    typename = basename.split("_")[0]
    if typename in self._knownTypes:
        return typename
    return "unknown"

My main justification that this only examines the bit of the file name that actually contains the type. It's no a big deal since the names are so simple. I used a class variable so the set only had to be constructed once, but that is an optimization that is a bit silly for so few values. It would be plenty fast enough to use if typename in ("bias", "flat", "defect"): and not even bother with a set.

If you prefer the simplicity of examining the whole basename, consider eliminating repetition of those names with a loop:

for typename in ("bias", "flat", "defect"):
    if typename in basename:
        return typename
return "unknown"

[r-owen]

Why is "expTime": "EXPTIME" necessary? Surely header name lookup is case blind?

[kfindeisen]

Columns must be present in translation or translators to be read at all. I tried testing the keywords just now and they are, surprisingly, case-sensitive.

(Technically I don't need the exposure time, but it's one of the few lsstSim columns that can be read without special code support so I figured I might as well leave it in.)

[kfindeisen]

The problem is that I'm not sure if I can add the right headers to the files and to assembleLsstChannels.py self-consistently, especially since I don't understand why the files have minimal headers in the first place (the script should have passed through all keywords from the original lsstSim data). Am I missing something obvious?

[parejkoj]

The original data probably just didn't have the missing headers.

[kfindeisen]

I don't think that's it. While we can't find the original data (another reason I'm reluctant to touch assembleLsstChannels.py), the test flats and biases in obs_lsstSim do have sensible-looking headers.

[parejkoj]

Hmm... it does look like the files in obs_lsstSim are of similar vintage. @SimonKrughoff : do you have any memory of this?

[r-owen]

I may have stripped the headers, though I'm surprised if so because it doesn't show up in the code.

I think adding OBSTYPE and a few things like that is perfectly safe. I would not change the format for any existing keywords.

[SimonKrughoff]

@kfindeisen asked about this before on slack. Unfortunately, those data are ancient and I don't know why they are different.