Use a neutral format to have lossless interface with JSON, Xarray, scipp, pandas

[loco-philippe]

What is the problem this feature will solve?

Each tool has a specific structure for processing multidimensional data with the following consequences:

• interfaces dedicated to each tool,
• partially processed data,
• no unified representation of data structures

Describe the desired outcome

The proposed format (see jupyter notebook, github repository, PyPI package ) is based on the following principles:

• neutral format available for tabular or multidimensional tools (e.g. Numpy, pandas, xarray, scipp, astropy),
• taking into account a wide variety of data types as defined in NTV format,
• high interoperability: reversible (lossless round-trip) interface with tabular or multidimensional tools,
• reversible and compact JSON format,
• Ease of sharing and exchanging multidimensional and tabular data,

Additional context

numpy/numpy#12481 (comment)

[github-actions]

Welcome to Astropy 👋 and thank you for your first issue!

A project member will respond to you as soon as possible; in the meantime, please double-check the guidelines for submitting issues and make sure you've provided the requested details.

GitHub issues in the Astropy repository are used to track bug reports and feature requests; If your issue poses a question about how to use Astropy, please instead raise your question in the Astropy Discourse user forum and close this issue.

If you feel that this issue has not been responded to in a timely manner, please send a message directly to the development mailing list. If the issue is urgent or sensitive in nature (e.g., a security vulnerability) please send an e-mail directly to the private e-mail feedback@astropy.org.

[pllim]

Based on the linked comment, looks like you specifically is targetting NDData only?

Is this related to the "Array API" stuff?

Hope you can clarify. Thanks!

[loco-philippe]

Thank you @pllim for your comment,

Considering Astropy, yes, NDData is well targeted (and table in the next version).

I'm not sure I understand the "Array API" stuff, if you think about the Python Array API standard, this initiative is not still included (but I think it should be).

[pllim]

Yes, that Array API. I just do not want this to be in conflict with the following issue, that might or might not be related:

• NDData should be compliant with numpy.array_api #15073