Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Enable downloads from AWS Open Data for all collections (#884)
* Ensure json schema fields are retained * Add direct s3 querying for regular docs * Fix timeout passing * Fix s3 threadsafety * Fix for latest data re-org * Use smart open * Fix deserialization performance * Switch to s3 for all doc queryies * Fix remaining type issues * More type fixes * Remove print * Fix deprecated field in materials and s3 * Fix type issues * Fix structure import * Fix open data json sanitization * Add projection to multithreaded s3 func * Remove default s3 query * Add comments for clarity in s3 func * Update pbar for s3 download * Switch grain boundary prefix * Fix s3 suffix delim * Fix task retrieval support * Linting * Docstring noqa * Add smart_open to deps * Linting * More linting * Add missing docstring * Docstring arg rename * Fix tests * Fix bs s3 query * Fix chgcar query * Fix dos query * More linting * Fix s3 keys * More s3 fixes * Last s3 query fix * Linting * Skip alloys generic test * Fix alloys skip * Remove task chemsys query * Linting * Fix similarity search name * More similarity fixes * Remove fermi rester * Remove fermi ref * More fermi fixes * Linting * Fix s3 decoding * Add deprecated filter for s3 * Fix object key pagination --------- Co-authored-by: Jason Munro <jasonmunro@Jasons-MBP.dhcp.lbl.gov>
- Loading branch information
Showing
13 changed files
with
433 additions
and
216 deletions.
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.