The meaning, history, and future of data_type

[d-chambers]

First, some history. data_type was one of DASCore’s earliest fixed-name attributes. Its original purpose was to distinguish between strain, strain-rate, and deformation-rate DAS data. Importantly, it predates the addition of unit support in DASCore.

The main question now is: in current DASCore, what is the semantic meaning of data_type, and how should it interact with data_units?

The issue is well illustrated by #511. After integrating strain_rate data, the data_type remains strain_rate, even though the units are updated correctly. Should integration inspect the free-form data_type string and decide whether to transform it, for example by dropping "rate"? That approach quickly becomes messy. Each function would need its own logic for interpreting and transforming arbitrary input strings.

Another approach is for each patch function to explicitly declare its output data_type, without trying to infer meaning from the input data_type. This is the approach taken in #693 and #681. It also motivated #702, which makes setting a data_type more ergonomic by allowing it to be specified in the patch_function decorator when a function has a single expected output type.

This approach is simple, but it can also cause problems. For example, with inverse operations such as integration and differentiation, setting integration output to something like "integrated_quantity" would mean differentiation should not blindly replace it with "differentiated_quantity".

More broadly, data_type may now be somewhat redundant. Patches already have units and history, which can convey the same information, although less directly.

So, going forward, should we:

1. Allow each function to set whatever data_type it wants?

2. If a function does not explicitly set data_type, should DASCore clear it to an empty string to avoid misleading metadata, as in patch integrate in TIME does not update data_type #511?

3. Should data_type be a strictly controlled vocabulary with documented semantics (eg the tuple in constants) or just free-form text?

4. Should we just deprecate data_type and rely on data_units and history to convey the same meaning?

[d-chambers]

this is what codex thinks:

I would lean toward deprecating data_type as semantic metadata, but not necessarily removing it immediately.

The clean model is:

• data_units describes the physical quantity.
• history describes how the data got there.
• data_type should not be required to remain physically accurate through transformations unless DASCore gives it formal semantics.

Right now data_type sounds semantic, but behaves like free-form text. That is the worst middle ground: users may trust it, but DASCore cannot reliably maintain it.

My recommendation:

1. Do not make every function parse and transform arbitrary data_type strings. That will become brittle quickly. Integration handling strain_rate -> strain is tempting, but then every operation needs special cases, aliases, plurals, domain-specific spellings, and inverse-operation logic.

2. If a function changes the physical meaning and does not explicitly set data_type, clear it. Keeping stale data_type="strain_rate" after integration is actively misleading. An empty or missing data_type is better than wrong metadata.

3. Keep allowing functions to explicitly set data_type, but treat it as advisory. For simple ingest or known outputs, this is useful. But it should not be considered authoritative compared with units.

4. Document that data_units is canonical for physical interpretation. data_type can remain a human-friendly label, legacy field, or optional classifier, but code should avoid depending on it for correctness.

5. Longer term, deprecate or rename it. If retained, something like data_category, measurement_type, or source_quantity might better communicate that it is a label, not a unit-aware semantic contract.

So for the listed options, my answer would be closest to:

• No to unrestricted “each function sets whatever it wants” as the main policy.
• Yes to clearing it when a function changes the data meaning and cannot confidently set a correct value.
• Probably yes to deprecating it as a semantic field, while keeping it around for compatibility.

The principle I would use is: wrong metadata is worse than missing metadata. data_type should only survive a transformation when DASCore has a well-defined reason to believe it is still true.

[d-chambers]

I am inclined to agree, but would like to hear other's thoughts.

[andreas-wuestefeld]

When I started with DAScore, the datatype was useful hint. Given all the complexity and transforms possible, I would never expect it to be trustworthy all along a processing chain.
I would rather have any function clear the data-type (or set to UNKNOWN) than trying to manage whatever a user might do with the data.

Following your questions:

1. Allow each function to set whatever data_type it wants?

yes

2. If a function does not explicitly set data_type, should DASCore clear it to an empty string to avoid misleading metadata,

yes

3. Should data_type be a strictly controlled vocabulary with documented semantics (eg the tuple in constants) or just free-form text?

users should be able to set it freely (and easily) in their homegrown code. Anything going into the main branch should restrict and manage allowed data_types

4. Should we just deprecate data_type and rely on data_units and history to convey the same meaning?

I find it a useful human-readable descriptor