Support Swath Data in CF

[ajelenak]

Title

Include Swath Data Encodings in the CF Document

Moderator

@erget

Moderator Status Review [last updated: 2020/05/15]

Awaiting a PR implementing the text changes to the Conventions. As stated below othis proposal has been reviewed

• at least twice by the CF Community at community meetings
• by CGMS WGI
  possibly by other groups as well. Thus the general approach seems good but of course the final text will need thorough review.

Requirement Summary

This proposal was presented at several past CF workshops during the course of its development. It has also been vetted by a number of subject matter experts. Given that it does not require any change of the data model or the conventions current text, it probably would fit best as a new appendix, similar to the current Appendix H for Discrete Sampling Geometries.

Technical Proposal Summary

Earth Science swath data originates as electromagnetic radiation collected from a specific direction into a solid angle and then measured at a number of electromagnetic spectrum intervals. The combination of the direction, the solid angle, and the instrument data acquisition settings defines one observation. At any given instant an instrument sweeps over an area of the Earth while its platform (an object carrying such instrument) moves. Successive observations are usually combined to cover a larger portion of the Earth. When these successive observations are plotted on maps they appear to cover a swath on the Earth’s surface, hence the name for this type of data. The proposed encodings are independent from the observation method and are applicable to swath data acquired by instruments on either satellites, airplanes, or unmanned aerial vehicles (UAV).

Benefits

All providers and users of remotely sensed geoscience data from satellites, airplanes, or UAVs.

Status Quo

Swath data remains unsupported by the CF conventions.

Detailed Proposal

The proposal is at https://github.com/Unidata/EC-netCDF-CF/blob/master/swath/swath.adoc. The new text for the conventions document will be based on the content in Sections 2.3 through 2.6.

[erget]

@ajelenak I think this is a good idea. Since nobody's volunteered yet I'm happy to moderate the discussion but it should be noted that I am clearly in favour of it so it would be more interesting to have somebody with more concerns. At a minimum I'd like to involve some people who can throw a cautious eye on it once you've got a PR ready.

[JonathanGregory]

I'd like to review this but I haven't had time yet. Thanks for compiling this carefully written proposal. Jonathan

[erget]

@ajelenak it looks like nobody's found glaring errors yet ;) would you mind putting together a PR proposing the changes to the text so that we can begin the discussion / approval process?

[hilawe]

@ajelenak this is an excellent and thorough effort. At NCEI we may have more examples for you to consider that could help with further definitions. I will let my colleagues know and see what they think, if it's not too late.

Thank you for your service to the community.

[gaochen-larc]

This is very helpful! Thank you @ajelenak !

For a follow-up question, I am dealing with data like the TRMM case. Should I use different groups to separate data with different dimensions? For example,

Group: lores
lat(time, samp_lo)
lon(time, samp_lo)

Group: hires
lat(time, samp_hi)
lon(time, samp_hi)

Any suggestions?

Thanks!

[lupemba]

@ajelenak
I just had a brief look at the document. The text seems focused on passive imagers. Is the Swath data format also applicable for radar data e.g. SAR or Scatterometers?

[davidhassell]

Hi - we have a new project starting in 2024 that will involves, as part of it, an extraction of data from climate and NWP models for comparison with L2 swaths, on the grid of the latter. Therefore my interest is piqued and I look forward to reviewing the proposal.

Thanks,
David

[ajelenak]

@lupemba Yes, SAR and scatterometer instruments were taken into account during development. Do you have a specific example we could test with?

[hilawe]

For a follow-up question, I am dealing with data like the TRMM case. Should I use different groups to separate data with different dimensions? For example,

Group: lores lat(time, samp_lo) lon(time, samp_lo)

Group: hires lat(time, samp_hi) lon(time, samp_hi)

@ajelenak I wanted to piggyback on this TRMM comment and provide a sample Passive Microwave Climate Data Record file that should cover what @gaochen-larc brought up. This should cover the maximum complexity of satellite swath files.

Thank you!

[lupemba]

@ajelenak,
The upcoming EPS-SG program is aiming to follow the CF format (where it is practicable). The test data can be found on this site and include test data for SCA (the new scatterometer) https://www.eumetsat.int/eps-sg-user-test-data
The backscatter in the test data is mostly noise and have a lot of missing values but the geometry of the swath should be close to what is expected for the real data.

I would recommend looking at SCA-1B-SZF and SCA-1B-SFR. The SZF is the full resolution where each beam has its own geometry. In the SZR the backscatter is resampled to produce 5 collocated measurements on one swath (two if you want to split the left and right side.).
Note that the swath is quite different for rotating fan beam scatterometers. I don't have any example of this.

[ajelenak]

@lupemba I got some sample SCA L1B files from the link you provided. The content in the SZF file /data group is in one of the possible swath formats. However, the content in the SZR file /data group is not. Below are several variables from that group in one SCA-1B-SZR file to illustrate how data are organized:

  group: data {
    dimensions:
      number_beams = 5;
      number_points = 3392;
    variables:
      double time(number_points=3392);
        :long_name = "time associated with each point";
        :units = "UTC seconds since 2020-01-01 00:00:00.000";

      int backscatter(number_points=3392, number_beams=5);
        :long_name = "backscatter coefficient (also known as NRCS or sigma0) obtained by spatial averaging the full resolution data around the grid point for the fore VV, mid VV, aft VV, mid HH and mid cross-pol channels";
        :units = "dB";
        :missing_value = -2147483648; // int
        :scale_factor = 1.0E-7; // double
        :add_offset = 0.0; // double

      int latitude(number_points=3392);
        :long_name = "geodetic latitude";
        :units = "degrees_north";
        :missing_value = -2147483648; // int
        :valid_min = -90000000; // int
        :valid_max = 89999999; // int
        :scale_factor = 1.0E-6; // double
        :add_offset = 0.0; // double

      int longitude(number_points=3392);
        :long_name = "longitude";
        :units = "degrees_east";
        :missing_value = -2147483648; // int
        :valid_min = -180000000; // int
        :valid_max = 179999999; // int
        :scale_factor = 1.0E-6; // double
        :add_offset = 0.0; // double
      uint line_index(number_points=3392);
        :long_name = "absolute grid index in along track";

      // ...

      uint line_index(number_points=3392);
        :long_name = "absolute grid index in along track";

      short node_index(number_points=3392);
        :long_name = "grid index in across track (far left swath to far right swath)";
        :valid_max = 53S; // short
        :valid_min = -53S; // short

      // ...
  }

This is not swath format because the backscatter, longitude, and latitude data are spatially stored as 1D. There is nothing wrong with this data organization but in CF this is very similar to a trajectory.