Skip to content

Change Scan Position Variable Type from Float to Int for IODA v2 Compliance#58

Merged
amstokely merged 4 commits intomainfrom
feature/scan_position_float_to_int
Mar 17, 2025
Merged

Change Scan Position Variable Type from Float to Int for IODA v2 Compliance#58
amstokely merged 4 commits intomainfrom
feature/scan_position_float_to_int

Conversation

@amstokely
Copy link
Collaborator

Summary:

This PR updates the scan position variable type from float to int to ensure compliance with the IODA v2 format.

Changes:

  • Modified the scan position variable type from float to int.

Motivation:

  • IODA v2 requires scan position to be stored as an integer rather than a floating-point value.

@amstokely amstokely requested review from jim-p-w and mgduda March 11, 2025 20:43
@amstokely amstokely self-assigned this Mar 11, 2025
@amstokely amstokely requested a review from st-ncar March 12, 2025 17:43
jim-p-w
jim-p-w previously approved these changes Mar 13, 2025
@st-ncar
Copy link
Collaborator

st-ncar commented Mar 14, 2025

I processed radiance observations using the current main and this feature branch and compared the resulting IODA files. It seems that the values of scan_position change as a result of the type conversion. For example, for the first 5 entries (the files in orig are the results of the main branch)

nccmp -ldmgefs -C 5 amsua_n15_obs_2018041500.h5 orig/amsua_n15_obs_2018041500.h5
DIFFER : VARIABLE : scan_position : TYPE : INT <> FLOAT
DIFFER : TYPES : ATTRIBUTE : _FillValue : VARIABLE : scan_position : INT <> FLOAT
DIFFER : VARIABLE : scan_position : POSITION : [0] : VALUES : -999 <> 20
DIFFER : VARIABLE : scan_position : POSITION : [1] : VALUES : -999 <> 21
DIFFER : VARIABLE : scan_position : POSITION : [2] : VALUES : -999 <> 22
DIFFER : VARIABLE : scan_position : POSITION : [3] : VALUES : -999 <> 23
DIFFER : VARIABLE : scan_position : POSITION : [4] : VALUES : -999 <> 24

Many scan_position entries seem to be assigned the _FillValue, others have entries that do not correspond to _FillValue but are different from the result in orig. Is this to be expected? I would have guessed that the numerical entries in the two files should be identical up to type conversion precision differences.

@amstokely
Copy link
Collaborator Author

I processed radiance observations using the current main and this feature branch and compared the resulting IODA files. It seems that the values of scan_position change as a result of the type conversion. For example, for the first 5 entries (the files in orig are the results of the main branch)

nccmp -ldmgefs -C 5 amsua_n15_obs_2018041500.h5 orig/amsua_n15_obs_2018041500.h5
DIFFER : VARIABLE : scan_position : TYPE : INT <> FLOAT
DIFFER : TYPES : ATTRIBUTE : _FillValue : VARIABLE : scan_position : INT <> FLOAT
DIFFER : VARIABLE : scan_position : POSITION : [0] : VALUES : -999 <> 20
DIFFER : VARIABLE : scan_position : POSITION : [1] : VALUES : -999 <> 21
DIFFER : VARIABLE : scan_position : POSITION : [2] : VALUES : -999 <> 22
DIFFER : VARIABLE : scan_position : POSITION : [3] : VALUES : -999 <> 23
DIFFER : VARIABLE : scan_position : POSITION : [4] : VALUES : -999 <> 24

Many scan_position entries seem to be assigned the _FillValue, others have entries that do not correspond to _FillValue but are different from the result in orig. Is this to be expected? I would have guessed that the numerical entries in the two files should be identical up to type conversion precision differences.

This is definiately not expected. I'll look into it right now. Thanks for testing out these changes.

@amstokely amstokely force-pushed the feature/scan_position_float_to_int branch from 2e3392a to 2ab69a3 Compare March 14, 2025 19:06
@amstokely
Copy link
Collaborator Author

I processed radiance observations using the current main and this feature branch and compared the resulting IODA files. It seems that the values of scan_position change as a result of the type conversion. For example, for the first 5 entries (the files in orig are the results of the main branch)

nccmp -ldmgefs -C 5 amsua_n15_obs_2018041500.h5 orig/amsua_n15_obs_2018041500.h5
DIFFER : VARIABLE : scan_position : TYPE : INT <> FLOAT
DIFFER : TYPES : ATTRIBUTE : _FillValue : VARIABLE : scan_position : INT <> FLOAT
DIFFER : VARIABLE : scan_position : POSITION : [0] : VALUES : -999 <> 20
DIFFER : VARIABLE : scan_position : POSITION : [1] : VALUES : -999 <> 21
DIFFER : VARIABLE : scan_position : POSITION : [2] : VALUES : -999 <> 22
DIFFER : VARIABLE : scan_position : POSITION : [3] : VALUES : -999 <> 23
DIFFER : VARIABLE : scan_position : POSITION : [4] : VALUES : -999 <> 24

Many scan_position entries seem to be assigned the _FillValue, others have entries that do not correspond to _FillValue but are different from the result in orig. Is this to be expected? I would have guessed that the numerical entries in the two files should be identical up to type conversion precision differences.

This is definiately not expected. I'll look into it right now. Thanks for testing out these changes.

I moved all scan_position type casting to write_obs, limiting changes to ncio_mod.f90 while ensuring the scan_position variable is correctly cast from real to integer. My previous approach would have required converting xdata%xseninfo_float to an integer array, which would have necessitated significant refactoring beyond ncio_mod and define_mod.

@amstokely amstokely requested a review from st-ncar March 14, 2025 20:51
st-ncar
st-ncar previously approved these changes Mar 14, 2025
Copy link
Collaborator

@st-ncar st-ncar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My comments have been addressed.

@amstokely amstokely requested a review from mgduda March 14, 2025 23:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants