Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve S3 documentation, testing and support #2686

Merged
merged 82 commits into from
May 11, 2023

Conversation

DennisHeimbigner
Copy link
Collaborator

@DennisHeimbigner DennisHeimbigner commented Apr 25, 2023

Improvements to S3 Documentation

  • Create a new document quickstart_paths.md that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths.
  • Modify nczarr.md to remove most of the S3 related text.
  • Move the S3 text from nczarr.md to a new document cloud.md.
  • Add some S3-related text to the byterange.md document.

Hopefully, this will make it easier for users to find the information they want.

Rebuild NCZarr Testing

In order to avoid problems with running make check in parallel, two changes were made:

  1. The nczarr_test test system was rebuilt. Now, for each test.
    any generated files are kept in a test-specific directory, isolated
    from all other test executions.
  2. Similarly, since the S3 test bucket is shared, any generated S3 objects
    are isolated using a test-specific key path.

Other S3 Related Changes

  • Add code to ensure that files created on S3 are reclaimed at end of testing.
  • Used the bash "trap" command to ensure S3 cleanup even if the test fails.
  • Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option --enable-s3 instead of --enable-nczarr-s3, although the latter is still kept as a deprecated alias for the former.
  • Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in.
  • Merge partial S3 support into dhttp.c.
  • Change the region defaulting mechanism to be as follows: if default is a defined profile, use that, otherwise use a fake profile no. The no profile used to be called none. It can only be used for non-authenticating requests.
  • Create an experimental s3 access library especially for use with Windows. It is enabled by using the options --enable-s3-internal (automake) or -DENABLE_S3_INTERNAL=ON (CMake). Also add a unit-test for it.
  • Move some definitions from ncrc.h to ncs3sdk.h

Other Changes

  • Provide a default implementation of strlcpy and move this and similar defaults into dmissing.c.

Addendum [5/9/23]

It turns out that attempting to test S3 using a github action secret is a very complex process. So, this was disabled for github actions. However, a new run_tests_s3.yml action file was added that will eventually encapsulate S3 testing.

## Improvements to S3 Documentation
* Create a new document *quickstart_paths.md* that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths.
* Modify *nczarr.md* to remove most of the S3 related text.
* Move the S3 text from *nczarr.md* to a new document *cloud.md*.
* Add some S3-related text to the *byterange.md* document.

Hopefully, this will make it easier for users to find the information they want.

## Rebuild NCZarr Testing
In order to avoid problems with running make check in parallel, two changes were made:
1. The *nczarr_test* test system was rebuilt. Now, for each test.
any generated files are kept in a test-specific directory, isolated
from all other test executions.
2. Similarly, since the S3 test bucket is shared, any generated S3 objects
are isolated using a test-specific key path.

## Other S3 Related Changes
* Add code to ensure that files created on S3 are reclaimed at end of testing.
* Used the bash "trap" command to ensure S3 cleanup even if the test fails.
* Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option *--enable-s3* instead of *--enable-nczarr-s3*, although the latter is still kept as a deprecated alias for the former.
* Get some of the github actions yml to work with S3; required fixing various test scripts adding a secret to access the Unidata S3 bucket.
* Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in.
* Merge partial S3 support into dhttp.c.
* Create an experimental s3 access library especially for use with Windows. It is enabled by using the options *--enable-s3-internal* (automake) or *-DENABLE_S3_INTERNAL=ON* (CMake). Also add a unit-test for it.
* Move some definitions from ncrc.h to ncs3sdk.h

## Other Changes
* Provide a default implementation of strlcpy and move this and similar defaults into *dmissing.c*.
It turns out that attempting to test S3 using a github action secret is a very complex process. So, this was disabled for github actions. However, a new *run_tests_s3.yml* action file was added that will eventually encapsulate S3 testing.
@DennisHeimbigner
Copy link
Collaborator Author

I have fixed the S3 testing in github actions. I have isolated the S3 testing to a new action file
name run_tests_s3.yml.
It should serve as the basis for the final fixes for doing manual invocation of S3 testing for github actions.

@WardF
Copy link
Member

WardF commented May 10, 2023

Thanks @DennisHeimbigner working through the rest of the review now.

@WardF WardF merged commit 4b2851d into Unidata:main May 11, 2023
98 checks passed
@DennisHeimbigner DennisHeimbigner deleted the s3update.dmh branch May 21, 2023 02:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants