Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve provenance information #1124

Merged
merged 5 commits into from
Sep 6, 2018
Merged

Improve provenance information #1124

merged 5 commits into from
Sep 6, 2018

Conversation

DennisHeimbigner
Copy link
Collaborator

There was a request to extend the provenance information
stored in the _NCProperties attribute to allow two things:

  1. capture of additional library dependencies (over and above
    hdf5)
  2. Recognition of non-netcdf libraries that create netcdf-4 format
    files.

To this end, the _NCProperties format has been extended to be
and arbitrary set of key=value pairs separated by commas.
This new format has version = 2, and uses commas as the pair separator.
Thus the general form is:
_NCProperties = "version=2,key1=value,key2=value2..." ;

This new version is accompanied by a new ./configure option of the form
--with-ncproperties="key1=value1,key2=value2..."
that specifies pairs to add to the _NCProperties attribute for all
files created with that netcdf library.

At this point, what is missing is some programmatic way to
specify either all the pairs or additional pairs
to the _NCProperties attribute. Not sure of the best way
to do this.

Builders using non-netcdf libraries can specify
whatever they want in the key value pairs (as long
as the version=2 is specified first).

By convention, the primary library is expected to be the
the first pair after the leading version=2 pair, but this
is convention only and is neither required nor enforced.

Related changes:

  1. Fixed the tests that check _NCProperties to properly operate with version=2.
  2. When reading a version 1 _NCProperties attribute, convert it to look
    like a version 2 attribute.
  3. Added some version 2 tests to ncdump/tst_fileinfo.c and
    ncdump/tst_fileinfo.sh

Misc Changes:

  1. Fix minor problem in ncdap_test/testurl.sh where a parameter to
    buildurl needed to be quoted.
  2. Minor fix to ncgen to swap switches -H and -h to be consistent
    with other utilities.
  3. Document the -M flag in nccopy usage() and the nccopy man page.
  4. Modify a test case to use the nccopy -M flag.

stored in the _NCProperties attribute to allow two things:
1. capture of additional library dependencies (over and above
   hdf5)
2. Recognition of non-netcdf libraries that create netcdf-4 format
   files.

To this end, the _NCProperties format has been extended to be
and arbitrary set of key=value pairs separated by commas.
This new format has version = 2, and uses commas as the pair separator.
Thus the general form is:
    _NCProperties = "version=2,key1=value,key2=value2..." ;

This new version is accompanied by a new ./configure option of the form
    --with-ncproperties="key1=value1,key2=value2..."
that specifies pairs to add to the _NCProperties attribute for all
files created with that netcdf library.

At this point, what is missing is some programmatic way to
specify either all the pairs or additional pairs
to the _NCProperties attribute. Not sure of the best way
to do this.

Builders using non-netcdf libraries can specify
whatever they want in the key value pairs (as long
as the version=2 is specified first).

By convention, the primary library is expected to be the
the first pair after the leading version=2 pair, but this
is convention only and is neither required nor enforced.

Related changes:
1. Fixed the tests that check _NCProperties to properly operate with version=2.
2. When reading a version 1 _NCProperties attribute, convert it to look
   like a version 2 attribute.
2. Added some version 2 tests to ncdump/tst_fileinfo.c and
   ncdump/tst_fileinfo.sh

Misc Changes:
1. Fix minor problem in ncdap_test/testurl.sh where a parameter to
   buildurl needed to be quoted.
2. Minor fix to ncgen to swap switches -H and -h to be consistent
   with other utilities.
3. Document the -M flag in nccopy usage() and the nccopy man page.
4. Modify a test case to use the nccopy -M flag.
@WardF WardF self-assigned this Aug 27, 2018
@WardF WardF added this to the 4.7.0 milestone Aug 27, 2018
@WardF
Copy link
Member

WardF commented Sep 6, 2018

I had to make a few edits to allow for tests failing when a different version of libhdf5 was on the test system than was on the system where some reference files were created. The issue appears resolved, and I'll merge this as soon as travis is finished.

@WardF WardF merged commit 3f2557e into master Sep 6, 2018
@WardF WardF deleted the provenance.dmh branch September 6, 2018 23:19
@DennisHeimbigner DennisHeimbigner restored the provenance.dmh branch November 16, 2018 20:31
@DennisHeimbigner DennisHeimbigner deleted the provenance.dmh branch November 16, 2018 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants