Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

preliminary AWS access #213

Merged
merged 18 commits into from
Oct 18, 2021
Merged

preliminary AWS access #213

merged 18 commits into from
Oct 18, 2021

Conversation

JessicaS11
Copy link
Member

There are currently a host of caveats to ICESat-2 cloud data access (including that it is NOT YET PUBLIC and STILL IN DEVELOPMENT), but this is a start towards some of the items that will be needed to enable it:

  1. an example notebook with the limitations and what you can do currently
  2. usage of a Query object to generate an s3 access token using your Earthdata credentials (assuming your Earthdata user ID will grant you access to the data for beta testing)
  3. usage of a granule search (within a query object) to generate s3 urls (with the caveat that these aren't yet included in granule metadata, so there's no way to filter out which urls are actually valid)

Acknowledging these caveats, I think we can move forward with reviewing this PR so that other beta testers can use the functionality, and we can continue improving cloud data access as we go.

@codecov-commenter
Copy link

codecov-commenter commented Jun 28, 2021

Codecov Report

Merging #213 (4f59bc0) into development (dc88afc) will decrease coverage by 0.48%.
The diff coverage is 22.72%.

Impacted file tree graph

@@               Coverage Diff               @@
##           development     #213      +/-   ##
===============================================
- Coverage        55.50%   55.02%   -0.49%     
===============================================
  Files               20       20              
  Lines             1526     1543      +17     
  Branches           317      321       +4     
===============================================
+ Hits               847      849       +2     
- Misses             622      636      +14     
- Partials            57       58       +1     
Impacted Files Coverage Δ
icepyx/core/query.py 39.06% <11.76%> (-2.52%) ⬇️
icepyx/core/granules.py 37.36% <60.00%> (+0.27%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dc88afc...4f59bc0. Read the comment docs.

JessicaS11 and others added 6 commits July 1, 2021 11:15
* create uml_action yml file

* debug

* fix path typo

* try a different action

first option tried was limited to master branch pushes

* add dot to args

* add class dot file from pyreverse

* update path to find dot files

* try different path to dot file

* run action to test graphviz install

* try different input file format

* add start and end flags to dot file

* Update uml_action.yml

* update path

* update path in second location

* add commit changes

* GitHub PlantUML generation auto-update

* remove graphvizdot flag

* GitHub PlantUML generation auto-update

* add graphviz testdot flag back

* switch to svg

* remove changes

* GitHub PlantUML generation auto-update

* try as a uml instead of dot

* try using pyreverse directly

* add package install

* try non-pip install of graphviz

* add sudo

* add requirements install

* ignore tests and install icepyx

* fix path to find file

* fix path to find file

* GitHub PlantUML generation auto-update

* try using current branch for commit

* remove plantuml and png files

* generate umls and remove extra files from testing

* update docs with new uml diagrams. Add visualization class to components list

* update action trigger to after approved pull request reviews

Co-authored-by: Romina Piunno <romina.piunno@gmail.com>
* change 'dataset' to 'product' in core and test code
* update 'source' input parameter to 'path' in variables module (part of a not-yet-implemented feature).
* update and run tests that require an active NSIDC Earthdata session
* add flat data resource note
* NOTE: using the fixture to successfully start a session can be considered the test
* transition options to json for cleaner test file
…openaltimetry visualization (#220)

Co-authored-by: Jessica <JessicaS11@users.noreply.github.com>
@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@github-actions
Copy link

github-actions bot commented Oct 13, 2021

Binder 👈 Launch a binder notebook on this branch for commit 6d76f24

I will automatically update this comment whenever this PR is modified

Binder 👈 Launch a binder notebook on this branch for commit 09cb18d

Copy link
Contributor

@betolink betolink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good!

icepyx/core/granules.py Show resolved Hide resolved
Copy link
Contributor

@betolink betolink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work Jessica, one thing to maybe take into account is that EGI subsetting is not available for cloud granules. I think the plan is that NASA Harmony will become the de facto subsetter for cloud collections. Right now ATL data is not supported by Harmony but when it does It should be fairly simple to integrate it with icepyx.

@JessicaS11 JessicaS11 merged commit 06610dc into development Oct 18, 2021
@JessicaS11 JessicaS11 deleted the aws_access branch October 18, 2021 16:30
JessicaS11 added a commit that referenced this pull request Dec 8, 2021
* add github action to add binder badge to PRs (#229)
* use the binder badge action directly (instead of a manual implementation of it) (#233)
See the discussion in #230 for more details on this switch.
* preliminary AWS access (#213)
* update links for travis badge (#234)
* Fix failing test_visualization_date_range check for ATL07 (#241)
* By default, no email status updates to users when ordering granules (#240)
* remove extra cell causing errors in example notebook
* Set default page size for orders to 2000 per NSIDC recommendation (#239)
* Add ICESat-2 data read-in functionality (#222)
* update examples from 2020 Hackweek tutorials
* update add and commit GitHub Action version (and UML diagrams) (#244)
* merge traffic (GitHub and PyPI) and bib updates (#245)
* Release 0.5.0 (#246)
* release v0.5.0 CI fixes (#251)
* fix Travis CI label in readme
* update earthdata login fixture for testing
* add required input to pytest fixture (#252)

Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>
Co-authored-by: trey-stafford <trey.stafford@colorado.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants