Update metadata handling for SigMF-NS-NTIA updates #83

aromanielloNTIA · 2023-06-14T19:01:48Z

Update metadata based on changes from Add diagnostics and data products. Specify sigan settings and cal info captures. JSON Schema 2020-12. sigmf-ns-ntia#42 for single frequency IQ, stepped frequency IQ, single frequency M4S, and NASCTN SEA actions
Bring sensor definition files closer in-line with the ntia-sensor specification. Now, they only differ by the fact that the sensor definition file may include a "location" object (which has not been altered). When the sensor definition is loaded, any location data is removed from the sensor definition dictionary and made available as a GeoJSON Point-formatted dict, SENSOR_LOCATION, for use elsewhere.
Add useful metadata generating base methods to abstract action classes
Metadata interfaces are created, leveraging msgspec. These may eventually roll into a dedicated Python package for sigmf-ns-ntia.
Make sigan and sensor calibration objects interchangeable. This is in line with how ntia-sensor views calibration objects: e.g., "gain" as a key name instead of "gain_sensor". Updated calibration example files to match.
Modify and add some capabilities on the base Action and MeasurementAction classes related to generating metadata
Update unit tests according to metadata changes

Tested M4S, single frequency IQ, stepped frequency IQ, and SEA data product actions on seadog02. Validated SigMF metadata JSON using the schema from sigmf-ns-ntia.

Note: This update will require a small change to sensor definition files. There should no longer be a top-level "id" field. Instead, use the "id" in "sensor_spec"

- use custom dataclasses for sigmf objects

dboulware

Very minor, but I would either move these up a level to the metadata package, or rename the interfaces package as they don't establish interfaces.

dboulware

Looks good. See in-line suggestions and comments.

scos_actions/actions/acquire_single_freq_fft.py

scos_actions/actions/acquire_stepped_freq_tdomain_iq.py

scos_actions/metadata/interfaces/ntia_sensor.py

scos_actions/metadata/interfaces/ntia_scos.py

scos_actions/metadata/interfaces/ntia_algorithm.py

dboulware

Looks good. Update version and it should be good to go.

will be included with #83

commit 92d7bda Merge: 7ede6cc 95de237 Author: Anthony Romaniello <66272872+aromanielloNTIA@users.noreply.github.com> Date: Thu Jun 22 14:06:47 2023 -0600 Merge pull request #83 from NTIA/update-sigmf-ns-ntia Update metadata handling for SigMF-NS-NTIA updates commit 95de237 Merge: 59e34ce 7ede6cc Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 22 16:05:22 2023 -0400 Merge branch 'master' into update-sigmf-ns-ntia commit 7ede6cc Merge: 6278878 0f357cf Author: Anthony Romaniello <66272872+aromanielloNTIA@users.noreply.github.com> Date: Thu Jun 22 14:04:24 2023 -0600 Merge pull request #81 from NTIA/patch-CVE-2023-32681 Bump `its_preselector` minimum version to patch CVE-2023-32681 commit 59e34ce Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 22 16:01:30 2023 -0400 bump version to 6.3.0 commit 0f357cf Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 22 16:00:43 2023 -0400 revert version bump will be included with #83 commit 7efd48e Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 22 12:49:10 2023 -0400 Make remaining required metadata fields non-optional commit 89c4f59 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 22 12:44:38 2023 -0400 RFPath "id" field should not be optional commit 9aa8835 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 22 12:42:21 2023 -0400 reset sigmf_builder for new captures in stepped-freq IQ commit e28bc94 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 22 12:36:10 2023 -0400 rename power detectors to statistical detectors commit 98aad41 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 22 12:09:47 2023 -0400 M4S action reference point from cal_adjust param commit ee77a17 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 22 12:07:06 2023 -0400 remove unused imports commit dc0a106 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Tue Jun 20 12:09:34 2023 -0400 rename metadata interfaces to metadata structs commit ea5b76e Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 15 15:04:07 2023 -0400 remove unused measurement_result values commit 9efb69b Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 15 15:01:34 2023 -0400 update metadata checks in tests commit a887684 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 15 14:55:14 2023 -0400 remove incorrect processing field from M4S graph commit b025891 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 15 14:15:39 2023 -0400 Add ntia-diagnostics extension metadata commit 8494b22 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 15 13:41:33 2023 -0400 convert x-axis Graph values to lists commit 1f70378 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 15 13:40:55 2023 -0400 Add DFT processing ID to M4S graph metadata commit 0c6f202 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 15 13:18:02 2023 -0400 Fix calibration metadata creation in stepped freq IQ commit c9ddcae Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 15 13:03:45 2023 -0400 fix calibration metadata build when cal data is None commit d0b0263 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 15 12:44:08 2023 -0400 Fix capturesegment kwarg commit e968509 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 15 12:41:41 2023 -0400 fix kwarg name commit bcdda24 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 15 12:40:42 2023 -0400 fix TypeError when cal data is None commit 336afa8 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 15 12:36:52 2023 -0400 use kwargs instead of posargs in create_capture_segment commit ddc76f3 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 15 12:15:11 2023 -0400 fix metadata values in PFP and PSD graphs commit e2f9516 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 15 11:13:47 2023 -0400 update example cal gain values commit 45c3b7e Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu Jun 15 10:58:38 2023 -0400 Add calibration and SiganSettings to captures also make sigan/sensor calibration data format consistent with SigMF extension commit 5ff10f8 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Wed Jun 14 19:23:13 2023 -0400 fix processing metadata type for M4S action commit 0cb6d96 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Wed Jun 14 18:53:29 2023 -0400 always build metadata inside getter avoids needing to call the builder in actions commit 732b2bf Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Wed Jun 14 18:35:53 2023 -0400 improve metadata building commit 6e622b7 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Wed Jun 14 17:50:44 2023 -0400 Fix metadata creation in MeasurementAction interface commit 0cbd647 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Wed Jun 14 15:30:47 2023 -0400 fix key rename commit fdce71c Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Wed Jun 14 15:00:55 2023 -0400 cleanup SEA action commit 0096390 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Wed Jun 14 14:46:09 2023 -0400 remove old debugging changes commit 706ce34 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Wed Jun 14 14:37:59 2023 -0400 Do not replace sensor ID with FQDN commit 5267d1b Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Wed Jun 14 14:32:18 2023 -0400 Revert making IQProcessor an instance variable commit b58a806 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Wed Jun 14 14:08:06 2023 -0400 Cleanup, add test todo's commit ee01c09 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Wed Jun 14 13:47:14 2023 -0400 clean up SEA metadata commit 6f7f40c Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Wed Jun 14 13:23:46 2023 -0400 fix dictionary loop commit 41edfe6 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Wed Jun 14 13:14:28 2023 -0400 handle captures and global separately when building commit 7227be6 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Wed Jun 14 12:55:11 2023 -0400 Update metadata builder to ensure objects serialize commit 37330a0 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Tue Jun 13 23:44:39 2023 -0400 custom JSON encoder handling of np.float64 values commit 8b0e4ff Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Tue Jun 13 23:18:13 2023 -0400 attempt fix capture segment serialization error commit 1e7d895 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Tue Jun 13 22:15:13 2023 -0400 fix add_capture object handling commit f11bc4e Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Tue Jun 13 21:42:51 2023 -0400 switch from dataclasses to msgspec commit 7aa7806 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Tue Jun 13 19:56:14 2023 -0400 fix enum not JSON serializable commit 12afee7 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Tue Jun 13 17:25:48 2023 -0400 fix keyname in computer diagnostics commit 56ee074 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Tue Jun 13 17:17:31 2023 -0400 fix keyerror commit 9ac7672 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Tue Jun 13 16:43:24 2023 -0400 add missing defaults for ntia-sensor commit 2b46355 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Tue Jun 13 16:34:53 2023 -0400 add missing len() commit 6485334 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Tue Jun 13 16:06:14 2023 -0400 update SEA action metadata builder commit 17fe5fc Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Tue Jun 13 16:05:17 2023 -0400 remove duplicate line commit bb65e9d Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Tue Jun 13 15:53:38 2023 -0400 fix sigmf_builder commit 2157675 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Tue Jun 13 11:38:08 2023 -0400 initial update for new sigmf-ns-ntia - use custom dataclasses for sigmf objects commit 2f39811 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu May 25 18:14:43 2023 -0400 bump patch version commit 8518d55 Author: Anthony Romaniello <aromaniello@ntia.gov> Date: Thu May 25 18:14:29 2023 -0400 bump preselector package to patched version

aromanielloNTIA added 22 commits June 13, 2023 11:38

initial update for new sigmf-ns-ntia

2157675

- use custom dataclasses for sigmf objects

fix sigmf_builder

bb65e9d

remove duplicate line

17fe5fc

update SEA action metadata builder

6485334

add missing len()

2b46355

add missing defaults for ntia-sensor

9ac7672

fix keyerror

56ee074

fix keyname in computer diagnostics

12afee7

fix enum not JSON serializable

7aa7806

switch from dataclasses to msgspec

f11bc4e

fix add_capture object handling

1e7d895

attempt fix capture segment serialization error

8b0e4ff

custom JSON encoder handling of np.float64 values

37330a0

Update metadata builder to ensure objects serialize

7227be6

handle captures and global separately when building

41edfe6

fix dictionary loop

6f7f40c

clean up SEA metadata

ee01c09

Cleanup, add test todo's

b58a806

Revert making IQProcessor an instance variable

5267d1b

Do not replace sensor ID with FQDN

706ce34

remove old debugging changes

0096390

cleanup SEA action

fdce71c

aromanielloNTIA added the enhancement New feature or request label Jun 14, 2023

aromanielloNTIA added this to the sea-prototype-v0.5 milestone Jun 14, 2023

aromanielloNTIA requested a review from dboulware June 14, 2023 19:01

aromanielloNTIA self-assigned this Jun 14, 2023

This was referenced Jun 14, 2023

Revisit metadata handling #40

Open

Update SCOS Actions for SigMF-NS-NTIA updates NTIA/scos-sensor#244

Merged

aromanielloNTIA added 2 commits June 14, 2023 15:30

fix key rename

0cbd647

Fix metadata creation in MeasurementAction interface

6e622b7

aromanielloNTIA added 7 commits June 15, 2023 13:18

Fix calibration metadata creation in stepped freq IQ

0c6f202

Add DFT processing ID to M4S graph metadata

1f70378

convert x-axis Graph values to lists

8494b22

Add ntia-diagnostics extension metadata

b025891

remove incorrect processing field from M4S graph

a887684

update metadata checks in tests

9efb69b

remove unused measurement_result values

ea5b76e

aromanielloNTIA marked this pull request as ready for review June 15, 2023 19:08

dboulware reviewed Jun 20, 2023

View reviewed changes

dboulware self-requested a review June 20, 2023 13:55

rename metadata interfaces to metadata structs

dc0a106

dboulware requested changes Jun 20, 2023

View reviewed changes

aromanielloNTIA added 6 commits June 22, 2023 12:07

remove unused imports

ee77a17

M4S action reference point from cal_adjust param

98aad41

rename power detectors to statistical detectors

e28bc94

reset sigmf_builder for new captures in stepped-freq IQ

9aa8835

RFPath "id" field should not be optional

89c4f59

Make remaining required metadata fields non-optional

7efd48e

aromanielloNTIA requested a review from dboulware June 22, 2023 16:53

dboulware requested changes Jun 22, 2023

View reviewed changes

aromanielloNTIA mentioned this pull request Jun 22, 2023

Bump its_preselector minimum version to patch CVE-2023-32681 #81

Merged

aromanielloNTIA added a commit that referenced this pull request Jun 22, 2023

revert version bump

0f357cf

will be included with #83

aromanielloNTIA added 2 commits June 22, 2023 16:01

bump version to 6.3.0

59e34ce

Merge branch 'master' into update-sigmf-ns-ntia

95de237

dboulware self-requested a review June 22, 2023 20:05

dboulware approved these changes Jun 22, 2023

View reviewed changes

aromanielloNTIA merged commit 92d7bda into master Jun 22, 2023
4 checks passed

aromanielloNTIA deleted the update-sigmf-ns-ntia branch June 22, 2023 20:06

aromanielloNTIA mentioned this pull request Jun 22, 2023

Update to SCOS Actions 6.3.0 NTIA/scos-tekrsa#44

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update metadata handling for SigMF-NS-NTIA updates #83

Update metadata handling for SigMF-NS-NTIA updates #83

aromanielloNTIA commented Jun 14, 2023 •

edited

dboulware left a comment

dboulware left a comment

dboulware left a comment

Update metadata handling for SigMF-NS-NTIA updates #83

Update metadata handling for SigMF-NS-NTIA updates #83

Conversation

aromanielloNTIA commented Jun 14, 2023 • edited

dboulware left a comment

Choose a reason for hiding this comment

dboulware left a comment

Choose a reason for hiding this comment

dboulware left a comment

Choose a reason for hiding this comment

aromanielloNTIA commented Jun 14, 2023 •

edited