how to best represent complex numbers and possibly other larger structures

[scottb77]

Hapi treats complex numbers as two dimensional real vectors for example the info request:
https://cdaweb.gsfc.nasa.gov/hapi/info?id=RBSP-B_WFR-SPECTRAL-MATRIX_EMFISIS-L2&parameters=BwEu, where information on the spectral power matrix component formed by the magnetic field w-antenna and electric field u-antenna measurements: BwEu is requested.

Returns

{
"HAPI": "2.0",
"status": {"code": 1200, "message": "OK"},
"parameters": [
{
"name": "Time",
"type": "isotime",
"units": "UTC",
"length":30,
"fill": null
},
{
"name": "BwEu",
"type": "double",
"units": "nT*V/m/Hz",
"fill": "-1.0E31",
"description": "---> [NO PLOTS] BwEu amplitude",
"size": [2,65],
"bins": [
{
"name": "Complex",
"units": " ",
"centers":[1.00e+00,2.00e+00]
},
{
"name": "WFR_frequencies_Fixed",
"units": "Hz",
"centers":[2.14e+00,4.27e+00,6.41e+00,8.55e+00,1.07e+01,1.28e+01,1.49e+01,1.71e+01,1.92e+01,2.14e+01,2.35e+01,2.56e+01,2.78e+01,3.10e+01,3.52e+01,3.95e+01,4.38e+01,4.91e+01,5.55e+01,6.30e+01,7.16e+01,8.01e+01,8.97e+01,1.00e+02,1.12e+02,1.26e+02,1.42e+02,1.59e+02,1.78e+02,2.00e+02,2.24e+02,2.52e+02,2.82e+02,3.16e+02,3.55e+02,3.98e+02,4.48e+02,5.02e+02,5.63e+02,6.31e+02,7.09e+02,7.96e+02,8.92e+02,1.00e+03,1.12e+03,1.26e+03,1.42e+03,1.59e+03,1.78e+03,2.00e+03,2.24e+03,2.52e+03,2.82e+03,3.17e+03,3.56e+03,3.99e+03,4.47e+03,5.02e+03,5.63e+03,6.32e+03,7.09e+03,7.96e+03,8.93e+03,1.00e+04,1.12e+04]
}
]
}
],
"startDate": "2012-09-01T00:00:04Z",
"stopDate": "2019-07-16T16:01:11Z",
"resourceURL": "https://cdaweb.gsfc.nasa.gov/misc/Notes.html#RBSP-B_WFR-SPECTRAL-MATRIX_EMFISIS-L2",
"contact": "voycrs@gmail.com"
}

The key "size" is [2,65] which is described by the key "bins" which indicates that the row is a complex number with 2 elements, of course, indicated by the Bins name "Complex", and the column is a frequency with 65 elements indicated by the Bins name "WFR_frequencies_Fixed", the server data stream is row major for this type of array. Currently the clients will return complex numbers as real 2D array or 2D sub-array and one must look at the “bins” name to determine if a dimension of size 2 of an array indicates it is a complex number.

[jvandegriff]

What about adding complex as a fundamental data type? This would involve adding complex-integer and complex-double to the type list. Any parameter of this type needs to have it's final dimension be of size 2, and those 2 elements contain the real and complex parts of the integer or double value.

(See the part of the spec that describes the current list of numeric types - it's near the beginning).

[jbfaden]

There was a discussion about this at today's telecon. I think this the complex type is one of a number of strange types which could appear within a grid of data. For example, you could have a grid of Time,Energy,3 where the 3 is for components theta, phi, and length. This is a good example to consider because theta and phi might have units of degrees, and the length is in volts.

[jvandegriff]

When an individual measurement quantity captured by a multi-dimensional parameter is also itself an array value (like a array of complex numbers or an array of 3-vector values), the dimension (or dimensions) associated with the measurement quantity is not going to have any binning associated with it. HAPI has no way to indicate what these extra dimensions mean, but you can say that there are no bins associated with a particular dimension by using:

"centers": null

in the bins object for any non-binned dimension.

For complex numbers, there would be a non-binned dimension of length 2 that holds the real and complex parts, and you would then just need a way to indicate which of the 2 elements was the real part and which the imaginary.

It would be difficult to come up with an enumerated list of possible meanings for non-binned dimensions. We could offer a way to interpret that dimension as an optional keyword,

This seems to point to the need for a more sophisticated way of describing multi-dimensional data, since right now we are assuming the dimensions are for 1) binned data or for 2) no reason (centers: null).

[jvandegriff]

Another simple idea for complex numbers would be to say that HAPI servers should always make a complex parameter's last dimension be the length=2 dimension that holds the real and complex values.

[scottb77]

For the rbsp set I verified that the complex numbers are returned as pairs when using hapi. I compared the .cdf data with the hapi data it was 100%. Scott From: jvandegriff ***@***.***> Reply-To: hapi-server/data-specification ***@***.***> Date: Monday, April 12, 2021 at 2:22 PM To: hapi-server/data-specification ***@***.***> Cc: "Boardsen, Scott A. (GSFC-674.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY]" ***@***.***>, Author ***@***.***> Subject: [EXTERNAL] Re: [hapi-server/data-specification] how to best represent complex numbers (#112) Another simple idea for complex numbers would be to say that HAPI servers should always make a complex parameter's last dimension be the length=2 dimension that holds the real and complex values.

[jvandegriff]

copied from now-closed duplicate ticket #111

How to represent:
complex numbers
uncertainties (linkages between a variable and uncertainties, like DELTA_PLUS and DELTA_MINUS in CDF files))
three-vectors (magnetic field, electric field, spacecraft position, etc)
larger semantic structures, like spectrum - there are different types

All of these require parameters to be linked to each other, similar to the way time varying bins are captured with a parameter name instead of the fixed bin centers or ranges.

[jvandegriff]

Need to solve in a way consistent with how we decide to describe or represent larger structures

[jvandegriff]

might need to push to 3.2 if too complocated

[jvandegriff]

Needs some examples: Scott can provide links to some CDF examples.

RBSP has some

[jvandegriff]

From CDAWeb the dataset ID rbsp-a_wfr-spectral-matrix_emfisis-l2

IDL> help, d.bubv.dat
FLOAT = Array[2, 65, 14400]
(it shows up in IDL as 2D array, not complex)

HAPI access to the data here:
https://cdaweb.gsfc.nasa.gov/hapi/info?id=RBSP-B_WFR-SPECTRAL-MATRIX_EMFISIS-L2

FYI, the SKT has corrected id:
https://cdaweb.gsfc.nasa.gov/pub/software/cdawlib/0SKELTABLES/rbsp-a_wfr-spectral-matrix_emfisis-l2_00000000_v01.skt