Resource Listing Distillation

Resource Listings in HAPI

This is a distillation of the previous wiki page on File Listings, which we now want to call Resource Listing.

The previous wiki had a lot of outdated and inconsistent info, and this page has just what I think we have decided to pursue, and is also designed to explain it in terms digestible by new contributors, and also by AI.

See also dataset schema issue tag.

See also https://github.com/hapi-server/data-specification/wiki/availability

It would be useful and enhance interoperability across data systems if certain types of responses from a HAPI server could be standardized. A schema that gives specific meaning to the contents of a HAPI dataset provides both semantic and syntactic surety to a client, enabling automated treatment of the HAPI content. Low hanging fruit for this kind of standardized expression are simple data types that are not measurements, but commonly experienced entities within a data system. This cinludes File Listings, Event Lists, and Data Availability Information, all of which are themselves a kind of meta-information about the content at a HAPI server (or other data providers, in the case of event lists).

This page summarizes our discussion and conclusions about how to implement a File Listing schema for HAPI that would allow a server to emit a standardized expression describing file names and other file properties. Think of it as a web-based, standardized expression for the Unis ls -l command. Many web servers provide file listings, and many programmers have written HTML scraping routines to grab these listings, but what if there could be a semantically solid, syntactically rigid way of expressing this? Any HAPI client coudl then get a file listing and know what it was dealing with. This woudl save so much time. Even for AI tools, the ability to use a standardized schema for common information will help automate routine tasks.

HAPI datasets can already represent a file listing. The stringType for a parameter will identify that parameter as a retrievable resource if the stringType is set to uri. An informal file listing is then just a dataset with at least a time and a URI. But many file listings include other details, and this document describes a not-quite-complete proposal create a schema that will describe and constrain what can be in an official, standardized file listing.

Events lists are more generally just a time stamp and a message associated with the time stamp. Often events lists will have a time range with a start and end time. There is a separate group working on an event list standard, and once that group has a schema, the HAPI team will be bery quick to create a HAPI-based expression of that standard. For now, this document does not address Event Lists.

Recommended dataset naming is to end it with "DATASET/filelist", e.g., "MyDataset/filelist" (not "filelist/MyDataset"), and also add "x_parameterSchema="filelist" in the info response.

One option is to start with a base schema class which is just a "Listing of Times":

Resource Listing Basics

The File Listing schema should capture basic file characteristics from a generic operating system, and make them avilable in a maximally interoperable way. The schema shoudl contain minimal reuiqred elements, but allow optional elements that cna provide more details about the files, especially those characteristics that are often needed when dealing with lists of scientific data files. The core items are of course the time and the file name, but here the time has a special scientific meaning, namely the start time of the data in the file. Becuase HAPI already has this concept, the label startDate will be used for this parameter. The URI parameter will just be uri. Optional but important items are modification date and size (both normal characteristics from the OS) and optional science-specific concepts are stopDate, checksum and version.

There was some debate about calling it a File Lsting or URI Listing and we setteld on Resource Listing since since we really aren't listing just plain files because this is a network-based access mechanism, which is focused on URIs instead of filesm and those URIs can point to a variety of resources, mostly files, but possibly other things.

Resource Listing Schema

So a Resource Listing schema needs to be able to describe time-ordereds list of data-centric URIs (files, S3 buckets, service endpoints, etc).

Note that the widest possible use of URIs is not supported here, because URIs can also capture mailto: and other non-data entities! This listing capability uses URI syntax, but only a subset of URIs are relevant for this kind of resource listing. The URI should point to an actionable, downloadable, data-centered content that is publicly accessible. The MIME type should indicate content that data users or systems could reasonably be expected to ingest.

The verifier could check for odd-ball URI types!

Resource Listing Schema

parameter name	parameter type	required / recommended / optional	description
startDate	isotime	required	start date/time of coverage
uri	string (and stringType of uri)	required	name of resource (file, image, endpoint, etc)
modificationDate	isotime	recommended	last modified time of the resource
size	number - see below	recommended	size of the file in B (bytes)
checksum	string (and stringType constrains the checkSumAlgorithm)	optional	value for checksum with the specified algorithm
creationDate	isotime	optional	time the resource was created
accessDate	isotime	optional	date & time the resource was last accessed
stopDate	isotime	optional	stop date & time of coverage; exclusive (like HAPI request end times)
version	string	optional	project-specific version indicator

About the size

Units must be in B and values formatted as an integer, unless the size is greater than 9,007,199,254,740,991 (~9 PB), in which case scientific notation should be used. Note that with this approach, a size is greater than ~9 PB cannot be represented with an exact number of bytes.

about version numbers

There are no constraints on the version strings. If there are multiple version of files over the life of a project, it may be desirable to offer a version indicator to distinguish between subsequent revisions (better calibration, background subtraction, etc).

Comment on column names: we tried to be consistent with the names used in the info response.

Ordering of "fileListing" columns:

• required columns must be present in the order given (startDate, then uri)
• recommended and/or optional columns must follow required columns and can be in any order
• any number of user-added columns can be present (other than the listed recommended / optional columns) and these can be interleaved among the optional columns

HAPI will need to define a new stringType for checksum (see ticket #273 in the HAPI specification GitHub repo). There are curated lists of hash algorithm names (on use case: HTTP headers). Consdier these lists of has algorithm names:

• https://www.iana.org/assignments/http-digest-hash-alg/http-digest-hash-alg.xhtml
  • CSV file with names: https://www.iana.org/assignments/http-digest-hash-alg/digest-fields.csv
• https://datatracker.ietf.org/doc/html/rfc9530

The first link has a CSV file with agreed on names - the algorithm name must be one of the items in the first column of that CSV file.

A long would be helpful here, but that should be a separate discussion (and maybe we also add float too, for HAPI 4.0; also complex numbers? The scope of adding more numeric types to HAPI is beyind the scope of this discussion so we are going with integer for now.

Events Lists are also listings and a basic representation coudl have these lements:

• startDate of time coverage (required,isotime)
• stopDate of time coverage (isotime, required) (Documentation acknowledges that this should be the same for an event that is just an instant, i.e., has no duration)
• label required string label or name for this event

Some example extensions to Event List:

• latitude
• longitude

There will be lots of extensions to the event list, and these will be domain-specific. The Solar communithy within Heliophysics already has a ery rich set of event characteristics for thigns that happen on (and now around) the Sun. See the event properties as described at the HEK site maintained by LMSAL.

Example proposed output, note x_parameterSchema

{
    "HAPI": "3.2",
    "x_createdAt": "2017-02-21T17:27Z",
    "modificationDate": "2026-01-01T00:00Z",
    "x_parameterSchema": "list>resourceList>jpgFileList",
    "parameters": [
        {
            "length": 20,
            "name": "startDate",
            "type": "isotime",
            "x_format": "$Y-$m-$dT$H:$M:$SZ",
            "fill": null,
            "units": "UTC",
            "timeStampLocation" : "begin"
        },
        {
            "description": "Picture of the creek, unmodified",
            "fill": null,
            "name": "uri",
            "length": 26,
            "type": "string",
            "units": null,
            "stringType": {
                "uri": {
                    "base": "https://cottagesystems.com/data/hapi/pics/",
                    "mediaType": "image/jpeg"
                }
            }
        },
        {
            "description": "File modification time",
            "name": "modificationDate",
            "type": "isotime",
            "fill": null,
            "x_format": "$Y-$m-$dT$H:$MZ",
            "length": 17,
            "units": "UTC"
        },
        {
            "description": "File size in kilobytes",
            "name": "size",
            "fill": null,
            "type": "integer",
            "units": "B"
        },
        { "name": "checksum",
         "type": "string",
         "length": 32,
         "stringType": {"checksum": { "algorithm": "md5" } },
         "fill": null,
         "description": "pre-calculated checksum using MD5 algorithm"}
    ],
    "sampleStartDate": "2023-01-01T00:00Z",
    "sampleStopDate": "2023-02-01T00:00Z",
    "startDate": "2022-11-01T00:00Z",
    "stopDate": "2026-03-06T00:00Z",
    "cadence": "PT10M",
    "status": {
        "code": 1200,
        "message": "OK"
    }
}

How to handle duration of files and events

How to handle the fact that event listing and file listings involve content that has an intrinsic time range. Regular HAPI data content has each row associated with a point in time, at least with respect to the query for data.

We decided to keep the query mechanism and rules the same, and will just add a statement about the need to expand a query time range to include potential edge cases, something like thies: "Because event lists and file listings refer to items with an implied durations, a HAPI query for items in this kind of list may need to be expanded, since the query will return only items whose start time falls in the query range. If a server wants to communicate a duration for a file or event, the stopDate should be used."

How to handle duplicate times in file listings or event lists

Repeated time tags are allowed in fileListing or eventList data schemas. Equivalently, we could say that data must never be decreasing.

We just noticed that the HAPI spec never actually states that HAPI times must only ever increase. So we need to add that to the spec! The definitions for "monotonically increasing: vary, so we will avoid that language. The spec shoudl say that values can only ever increase, with no duplicates.

Comments on case and capitalization

Three places where we have specific capitalization:

• http query parameters: we use snake case, such as include_parameters
• camelCase everywhere else
• AlertCamelCase for the name of the first column, the Time parameter (sort of, since it's only one word)

Defining the schema for what the parameters are

Like the unitsSchema and coordinateSystemSchema, we will use parameterSchema as the keyword in the HAPI info response.

Other options considered:

• datasetSchema - this means keywords outside the parameters have extra requirements; could datsetSchema be an array?
• dataType - Should it just be called dataType? Do we need to worry about other usage of schema? We have stringType already.

We settled on datasetSchema, since this ties the schema to the whole dataset, not just the parameters and declaring a specific datasetSchema may actually then levy requirements on other items in the info response at the dataset level. For example, if we add a schema for ground-based datasets, we could require the location or geoLocation.

List of potential schemas:

• resourceListing
• FAIR
• eventList
• groundMagnetometer (and other possible measurement types)
• spaceMagnetometer
• energeticParticles
• radioSpectrum
• plasmaParameters
• streamFlow
• earthquakeSeismograph
• salinityTrace

What about multiple schemas in one info response? We should design the current mechanism to allow for this in the future, but not include it now.

What about the ability to map non-standard names of parameters into the standard ones needed to fulfill a specific datasetSchema? After discussion, we realized that HAPI server creators control the dataset level keyword names, so if a server wants to be compliant with a certain datasetSchema, that server shoudl do the mapping of the names internally, and not express that mapping in HAPI info metadata. This saves HAPI from having to include syntax for renaming things, which if the server knows about that, it could just do it anyway. Although, having the name mapping block in HAPI woudl allow existing dataset to be merely augmented in a backward compatible way to become compliant with a new datasetSchema, and old code could still refere to the legacy names of the parameters and new code could leverage the name mapping to grab the standardized (via the datasetSchema) version of the data. This may still be worth considering.

The name mapping element could look like some of these options. Note that this is a mishmash info response with multiple illustrations of possibilities for the mapping (including no mapping), whereas a real info reponse woudl just have one mapping (or none). And it has comments (using #) which are illegal in real JSON.

{
   "HAPI": "3.3",
   "status": { "code": 1200, "message": "OK"},
   "startDate": "1998-001Z",
   "stopDate" : "2017-100Z",
   "coordinateSystemSchema" : { "schemaName": "SPASE", "schemaURI": "TBD"},
   "datasetSchema": "fileListing", # no name mapping needed!
   "datasetSchema": { "fileListing": {} },  # allowed?, but same as above
   "datasetSchema": { "fileListing": { "parameterMap" : { "startDate" : "Time", "uri": "fileURL" } } },
   # or for data:
   "datasetSchema": { "groundMagnetometer": { "vectorField" : "field" } },
   "datasetSchema": { "groundMagnetometer": { "vectorField" : [ "bx", "by", "bz"] } },
   "datasetSchema": { "groundMagnetometer": { "vectorFieldBaselineSubtracted" : "fieldBGSubtr" } },
   "datasetSchema": { "groundMagnetometer": { "vectorFieldBaselineSubtracted" : [ "bx_b", "by_b", "bz_b"] } },
}