Conversation
Codecov Report
❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more. @@ Coverage Diff @@
## main #342 +/- ##
=======================================
Coverage ? 70.35%
=======================================
Files ? 9
Lines ? 506
Branches ? 0
=======================================
Hits ? 356
Misses ? 150
Partials ? 0 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
ascourtas
added a commit
that referenced
this pull request
Nov 14, 2023
* update to 0.6.0 for HTTPS pub * Upload Foundry class load() function to default download using https (#340) * Update setup.py Fix version number for pyPI deploy * Update setup.py version for pyPI * Update requirements.txt to latest DLHub SDK This is needed to require upgrade of DLHub SDK for Foundry users when they upgrade Foundry. * Update version to 0.6.3 * incorporate load() to foundry.__init__() * automating api documentation using github action (#342) * CI: Automated documentation build * Removing remnants of XTract * CI: Automated documentation build * CI: Automated documentation build * Update README.md with contributing instructions (#357) * Update README.md with contributing instructions * Update PR language * merging in split specification * flake fixes * add jingrui examples (#363) * removed blank line * Load on init (#358) * incorporate load() to foundry.__init__() * merging in split specification * flake fixes * removed blank line * Adds note for quickstart globus set to false * Validating metadata before publishing * remove arguments from Foundry object that are duplicated with base class * Update setup.py version to 0.7.0 * refactor foundry to separate foundry instance from dataset objects * fine tuning search functionality * removing redefinition of FoundryDataset * address comments in PR review * remove unused import * updgrade setup-python from v2 to v4 * upgrade other setup-python from v2 to v4 * modify limit test --------- Co-authored-by: ascourtas <ascourtas@gmail.com> Co-authored-by: Ben Blaiszik <blaiszik@uchicago.edu> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Marshall McDonnell <mcdonnellmt@ornl.gov>
blue442
added a commit
that referenced
this pull request
Jan 19, 2024
* update to 0.6.0 for HTTPS pub * Upload Foundry class load() function to default download using https (#340) * Update setup.py Fix version number for pyPI deploy * Update setup.py version for pyPI * Update requirements.txt to latest DLHub SDK This is needed to require upgrade of DLHub SDK for Foundry users when they upgrade Foundry. * Update version to 0.6.3 * incorporate load() to foundry.__init__() * automating api documentation using github action (#342) * CI: Automated documentation build * Removing remnants of XTract * CI: Automated documentation build * CI: Automated documentation build * Update README.md with contributing instructions (#357) * Update README.md with contributing instructions * Update PR language * merging in split specification * flake fixes * add jingrui examples (#363) * removed blank line * Load on init (#358) * incorporate load() to foundry.__init__() * merging in split specification * flake fixes * removed blank line * Adds note for quickstart globus set to false * Validating metadata before publishing * remove arguments from Foundry object that are duplicated with base class * Update setup.py version to 0.7.0 * refactor foundry to separate foundry instance from dataset objects * fine tuning search functionality * removing redefinition of FoundryDataset * address comments in PR review * remove unused import * updgrade setup-python from v2 to v4 * upgrade other setup-python from v2 to v4 * modify limit test --------- Co-authored-by: ascourtas <ascourtas@gmail.com> Co-authored-by: Ben Blaiszik <blaiszik@uchicago.edu> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Marshall McDonnell <mcdonnellmt@ornl.gov>
blue442
added a commit
that referenced
this pull request
Jan 26, 2024
* fixed atom finding notebook * fixed atom position finding notebook * Repo visualizer: updated diagram * Adding Zeolite OSDB example * Adding solvation energy example * updated zeolite notebook * Repo visualizer: updated diagram * found the .list() return in the code. just need to use a debugger to see actual data object * added DOI to list() method * added DOI to the list() description * added DOI to _repr_html_ need to preview it but pushing the addition for now * changed formatting a bit by adding a label for DOI * made DOI a <p> element for better spacing * replace spaces with tabs in notebooks * Repo visualizer: updated diagram * establish run() function again * add dlhub_sdk==0.10.0 to reqs, and update foundry version to 0.1.2 * delete extraneous old requirements.txt * bound globus-sdk to <=2.0.3, since Foundry isn't compatible with Globus SDK 3 yet * add dlhub to list of service used in testing * fix syntax typo in service listing * add funcx to list of services * Updating auth for tests and client creation * Repo visualizer: updated diagram * deleting dupe of python-publish.yml * add auth handling to init if authorizers not provided * update local tests to work with auth for search * remove passing decorators for tests that are local * add 'mdf' as the default index * update tests to use prod dataset * update reqs for Globus SDK 3, and version to 0.2.0 * Repo visualizer: updated diagram * add option to pass in funcx_endpoint in run() * add pip dependecy caching * test caching functionality * Add keep_hdf5 functionality to Foundry.load_data * Add Convert to Pytorch Dataset Functionality * Add Testing for toTorch function * Remove unnecessary imports * Replace keep_hdf5 with as_hdf5 * Update foundry/foundry.py * Update foundry/foundry.py * Update foundry/foundry.py * Replace keep_hdf5 with as_hdf5 * Repo visualizer: updated diagram * Update Testing and Rename Files * Fix Testing Errors * Fix a singular typo * Replace PNG with SVG * Rebase with dev * Rebase with dev * Fix first set of TODOs * Finish TODO's for Load * Fix Dataset Loading * Apply Logan's Changes * Fix Logging and Remove unnecessary code * Fix Logging and Remove unnecessary code * Add Ari's Requests. * fix unused imports, code style, and update syntax (#229) * Replace keep_hdf5 with as_hdf5 * Update foundry/foundry.py * Update foundry/foundry.py * Update foundry/foundry.py * Repo visualizer: updated diagram * fix unused imports, code style, and update syntax Co-authored-by: Aadit Ambadkar <58674441+Aadit-Ambadkar@users.noreply.github.com> Co-authored-by: Ben Blaiszik <blaiszik@uchicago.edu> Co-authored-by: repo-visualizer <repo-visualizer@users.noreply.github.com> * re-add improperly removed warnings import * add Python 3.10 tests and flake8 error checking * add setup.cfg to change flake8 parameters * add comments so flake8 ignores 'unused import' err * merge tests into single file (#233) * merge tests into single file * change name of test file to match new commit * convert is_gha to a boolean for pytest skip * remove premature optimization * Reflect Changes * Fix Try Catch and Logging.log * Fix Logging * Make Logger Reflect Module Name * add dl.easy_publish wrapper function * Imports * Revert "add dl.easy_publish wrapper function" This reverts commit 7f83611. * add dl.easy_publish wrapper as f.publish_model * remove commented code and link to dlhub docs in docstring * Update testing-work.yml * Repo visualizer: updated diagram * Update README.md * Repo visualizer: updated diagram * Fix Logging * update test name * Rename testing-work.yml to tests.yml * Rapid removing of XTract (#242) * Rapid removing of XTract * Fixing as_object * Repo visualizer: updated diagram * Update setup.py * Repo visualizer: updated diagram * Update setup.py * Repo visualizer: updated diagram * end the file with a newline * remove redundant flake8 checking * fix code style (without changing functionality) * add more style fixes * fix last style error, others are covered in #231 * To tf dataset (#201) with rebase * Add Custom Dataset and Implement * Clean up Branch * Clean up Branch * Resolve Some of Logan's Changes * Resolve Testing Issues? * Resolve Testing Issues? * Resolve Testing Issues? * Resolve Testing Issues? * Resolve Testing Issues? * Reflect Logan's Requests * Fix Import Issues * Simplify Imports * Fix Imports * Apply Logan's Changes * Comments * Refactor Common Logic Into New Function * Add Documentation * Add Documentation * Add Custom Dataset and Implement * Clean up Branch * Replace keep_hdf5 with as_hdf5 * Resolve Some of Logan's Changes * Resolve Testing Issues? * Resolve Testing Issues? * Resolve Testing Issues? * Resolve Testing Issues? * Resolve Testing Issues? * Reflect Logan's Requests * Fix Import Issues * Simplify Imports * Fix Imports * Apply Logan's Changes * Comments * Refactor Common Logic Into New Function * Add Documentation * Add Documentation * fix reference to _get_inputs_to_targets(); also, whitespace * remove unused * import * fix test_foundry.py to have the proper tests from the dev branch * remove outdated test_to_pytorch() test * fix passing of self for _get_inputs_targets() Co-authored-by: Aristana Scourtas <ascourtas@gmail.com> * delete deprecated build() function and remove 'fail' language from tabular dataset reading * remove redundant path checking code for loading datasets * fix logic error in data path verification * break out path joining logic to be in scope for all dataset types * add new easy_publish parameters * set defaults for new parameters * update to version 0.3.0, add reqs for dlhub 1.0.0, update PyPI info * Repo visualizer: updated diagram * Delete bubble-vis.yml * Update README.md * Update README.md * Add files via upload * Update README.md * Add files via upload * Update README.md * address flake8 concerns in foundry.py * fix flake8 concerns in torch_wrapper * address flake8 concers for tf_wrapper * replace xtract module name with https one in __init__.py * transfer https methods to https_download.py * reorder private method * final flake8 changes * fix import of https_download * Add search functionality to make it easier to find datasets * Flake8 fixes * Data packages --> datasets * Dev (#255) * Fix Logging and Remove unnecessary code * Fix Logging and Remove unnecessary code * add Python 3.10 tests and flake8 error checking * add setup.cfg to change flake8 parameters * add comments so flake8 ignores 'unused import' err * Reflect Changes * Fix Try Catch and Logging.log * Fix Logging * Make Logger Reflect Module Name * Imports * Fix Logging * update test name * Rename testing-work.yml to tests.yml * end the file with a newline * remove redundant flake8 checking * fix code style (without changing functionality) * add more style fixes * fix last style error, others are covered in #231 * delete deprecated build() function and remove 'fail' language from tabular dataset reading * remove redundant path checking code for loading datasets * fix logic error in data path verification * break out path joining logic to be in scope for all dataset types * address flake8 concerns in foundry.py * fix flake8 concerns in torch_wrapper * address flake8 concers for tf_wrapper * replace xtract module name with https one in __init__.py * transfer https methods to https_download.py * reorder private method * final flake8 changes * fix import of https_download * Add search functionality to make it easier to find datasets * Flake8 fixes * Data packages --> datasets Co-authored-by: Aadit-Ambadkar <aadit.ambadkar@gmail.com> Co-authored-by: Isaac Darling <61167692+isaac-darling@users.noreply.github.com> Co-authored-by: Aadit Ambadkar <58674441+Aadit-Ambadkar@users.noreply.github.com> Co-authored-by: Ben Blaiszik <blaiszik@uchicago.edu> * update version to 0.4.0 also, add Braeden as contributor * update version * Fix test badge * Create README.md * Open in Colab Buttons (#253) * Repo visualizer: updated diagram * Add Badges * Update Positioning Co-authored-by: ascourtas <ascourtas@gmail.com> Co-authored-by: repo-visualizer <repo-visualizer@users.noreply.github.com> * Moving some functions to utils (#262) * Moving some functions to utils * flake8 fixes * Update README.md added more to the examples readme to make it more inviting/give better context as to what this page is * Update README.md (#270) added screenshots to readme with text explainations * Update read logic (#271) * initial example for QMC ML * simplifying read logic * swap testing dataset for a smaller dataset * Added new search test. Removed some stray commented-out code. (#272) * Paralellize HTTPS downloads. Remove joblib and six requirements (#273) * Get citation function (#274) * Paralellize HTTPS downloads. Remove joblib and six requirements * Add initial bibtex citation output function * update version to 0.5.0 * Improve HTTPS downloads (#277) * Cleaned up the keyword arguments, docstring - Only one keyword argument is used, so having a large (and undocumented) flexibility with **kwargs is unneeded - Docstring style mixed NumPy and Google * Add a test requirements file, simplify test YAML Sorry, a little house maintenance while I'm at it * Fix how parallel downloads are implemented Previous version was using an executor in a way that would produce an unlimited number of threads, which can cause problems for large datasets * Make a progress bar, error checking * Removed deprecated code It will live in git and our hearts forever * Flake8 fixes * Update README.md (#275) * Add NSF badge to Foundry * updated model pub notebook (#284) * Set Header Images to an absolute URL via raw.githubusercontent (#283) * Update links to absolute URL for pypi visibility * Redirect URL to MLMI2-CSSI * Update README.md (#299) * Add new logo * Add updated logos * Update README [no ci] * Updating example logos * Remove stray print _read_json was printing a debug data frame. Removed. * Add web (#301) * Increment version for PyPI deploy * Update issue templates * Add https upload (#281) * add initial directory-making functionality * add acl permission setting * add PUT request logiv and ACL setting, plus TODOs * add logic to delete acl rule after creation * add try/except handling to acl creation * add prepare query param so we don't need to make dirs; fix bug when rule_id is not set * clean up path joining logic, as well as comments * add capability to upload all files in a folder, instead of one individual file * update endpoint destination to use a UUID as the folder name * break out acl rule adding to its own function, tidy up * break out PUT request functionality * break out upload_folder() into upload_file() and integrate https functions into publish(), with proper params * change endpoint to NCSA, make usage more modular; small os.path bug fixes * reorder functions to be easier to read * add upload capability for single file, with error handling * fix logic bugs with destination path setting s.t. all subfolders are written to destination * cleanup var names in upload_folder() logic; making endpoint_dest path more robust * code cleanup and breakout helper functions to reduce size of publish() * add parameter checks to publish() and reduce param complexity * add docstrings, plus add test param to publish() * appease flake8 * add one more flake8 fix * fix auths in tests, add system test for HTTPS publication, small comments * add system test for HTTPS upload * break out https publishing into more unit-testable method * refactor function defs to work better for testing; add https upload unit test * fix bug where artifact was written to uploaded dataset * update os.walk block comparison to be more robust * update publish() docstring and add type hints * clean up imports, fix type hint for Response, add some context for Xtract file * WIP to separate helpers into submodule -- need to fix test and method design * fix typing discrepancy for requests.Response * update modification date * Temporarily remove ACL rule creation for https upload * Fix flake8 comment error * Fix flake8 once more * Fixing local tests, flake8, kwargs * Adding test data * Debug result on GHA * Debug result on GHA * Debug result on GHA * Debug result on GHA * add Ben's patch to submodule * generalize the included functions and move make_globus_link here from foundry object * move make_globus_link function to submodule * update tests to generalized input format * properly pass 'auths' object between functions * update modification date * prepend underscore to private function * correct call to upload_to_endpoint() in foundry.py * re-add ACL rule logic * update auth passing to be more user-friendly; includes test changes * Introduce a collection to hold authorizers It uses a dataclass so that we can annotate the type of authorizers that the tuple, then document them I put it in a new module, `foundry.auth` so that it can be used by both the foundry module and the https_upload module (avoiding circular dependencies) * alter args such that it's not possible for the user to have endpoint_id and gcs_auth_client misalign * change language to endpoint_auth_clients for clarity of purpose * docstring updates --------- Co-authored-by: Ben Blaiszik <blaiszik@uchicago.edu> Co-authored-by: isaac-darling <61167692+isaac-darling@users.noreply.github.com> Co-authored-by: Logan Ward <ward.logan.t@gmail.com> * https upload bugfix (#322) * add initial directory-making functionality * add acl permission setting * add PUT request logiv and ACL setting, plus TODOs * add logic to delete acl rule after creation * add try/except handling to acl creation * add prepare query param so we don't need to make dirs; fix bug when rule_id is not set * clean up path joining logic, as well as comments * add capability to upload all files in a folder, instead of one individual file * update endpoint destination to use a UUID as the folder name * break out acl rule adding to its own function, tidy up * break out PUT request functionality * break out upload_folder() into upload_file() and integrate https functions into publish(), with proper params * change endpoint to NCSA, make usage more modular; small os.path bug fixes * reorder functions to be easier to read * add upload capability for single file, with error handling * fix logic bugs with destination path setting s.t. all subfolders are written to destination * cleanup var names in upload_folder() logic; making endpoint_dest path more robust * code cleanup and breakout helper functions to reduce size of publish() * add parameter checks to publish() and reduce param complexity * add docstrings, plus add test param to publish() * appease flake8 * add one more flake8 fix * fix auths in tests, add system test for HTTPS publication, small comments * add system test for HTTPS upload * break out https publishing into more unit-testable method * refactor function defs to work better for testing; add https upload unit test * fix bug where artifact was written to uploaded dataset * update os.walk block comparison to be more robust * update publish() docstring and add type hints * clean up imports, fix type hint for Response, add some context for Xtract file * WIP to separate helpers into submodule -- need to fix test and method design * fix typing discrepancy for requests.Response * update modification date * Temporarily remove ACL rule creation for https upload * Fix flake8 comment error * Fix flake8 once more * Fixing local tests, flake8, kwargs * Adding test data * Debug result on GHA * Debug result on GHA * Debug result on GHA * Debug result on GHA * add Ben's patch to submodule * generalize the included functions and move make_globus_link here from foundry object * move make_globus_link function to submodule * update tests to generalized input format * properly pass 'auths' object between functions * update modification date * prepend underscore to private function * correct call to upload_to_endpoint() in foundry.py * re-add ACL rule logic * update auth passing to be more user-friendly; includes test changes * Introduce a collection to hold authorizers It uses a dataclass so that we can annotate the type of authorizers that the tuple, then document them I put it in a new module, `foundry.auth` so that it can be used by both the foundry module and the https_upload module (avoiding circular dependencies) * alter args such that it's not possible for the user to have endpoint_id and gcs_auth_client misalign * change language to endpoint_auth_clients for clarity of purpose * docstring updates * fix bug from last round of review edits --------- Co-authored-by: Ben Blaiszik <blaiszik@uchicago.edu> Co-authored-by: isaac-darling <61167692+isaac-darling@users.noreply.github.com> Co-authored-by: Logan Ward <ward.logan.t@gmail.com> * add static badge with link to gitbook (#333) * Update publishing notebook and minor bugfixes (#336) * update publishing notebook example to use HTTPS upload primarily, along with minor fixes * add https upload methods and data * fix function call to publish_dataset * remove ACL rule code to fix error issue * update globus images in notebook * remove commented code * add missing scopes * appease flake overlords * Add search lambda authorizer (sl_authorizer) to dlhub_client instantiation * removed unnecessary scopes * update curation info in notebook --------- Co-authored-by: Ben Blaiszik <blaiszik@uchicago.edu> Co-authored-by: isaac-darling <61167692+isaac-darling@users.noreply.github.com> Co-authored-by: Logan Ward <ward.logan.t@gmail.com> Co-authored-by: Eric Blau <blau@anl.gov> * delete due to unresolvable merge conflict -- adding back in dev * add back publishing notebook * Split specification (#344) * adding ability to specify splits for loading * refining test * Update splits_to_load --> splits --------- Co-authored-by: blaiszik <blaiszik@uchicago.edu> * Implementing new search() function (and refactor) (#408) * update to 0.6.0 for HTTPS pub * Upload Foundry class load() function to default download using https (#340) * Update setup.py Fix version number for pyPI deploy * Update setup.py version for pyPI * Update requirements.txt to latest DLHub SDK This is needed to require upgrade of DLHub SDK for Foundry users when they upgrade Foundry. * Update version to 0.6.3 * incorporate load() to foundry.__init__() * automating api documentation using github action (#342) * CI: Automated documentation build * Removing remnants of XTract * CI: Automated documentation build * CI: Automated documentation build * Update README.md with contributing instructions (#357) * Update README.md with contributing instructions * Update PR language * merging in split specification * flake fixes * add jingrui examples (#363) * removed blank line * Load on init (#358) * incorporate load() to foundry.__init__() * merging in split specification * flake fixes * removed blank line * Adds note for quickstart globus set to false * Validating metadata before publishing * remove arguments from Foundry object that are duplicated with base class * Update setup.py version to 0.7.0 * refactor foundry to separate foundry instance from dataset objects * fine tuning search functionality * removing redefinition of FoundryDataset * address comments in PR review * remove unused import * updgrade setup-python from v2 to v4 * upgrade other setup-python from v2 to v4 * modify limit test --------- Co-authored-by: ascourtas <ascourtas@gmail.com> Co-authored-by: Ben Blaiszik <blaiszik@uchicago.edu> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Marshall McDonnell <mcdonnellmt@ornl.gov> * CI: Automated documentation build * getting cache sorted * getting cache sorted * updated docstrings and examples, added extended pandas dataframe class for display and access of datasets * updated docstrings and examples, added extended pandas dataframe class for display and access of datasets * cleaning up flake8 issues * cleaning up flake8 issues * cleaning up a few loose ends * cleaning up a few loose ends * check if download is causing test in GHA to fail * check if download is causing test in GHA to fail * disabling all individual tests * disabling all individual tests * testing a few tests * testing a few tests * testing a few more tests * testing a few more tests * testing w/ globus=False * testing w/ globus=False * WIP * WIP * refactoring tests * refactoring tests * wrapping up testing of foundry_cache.py * wrapping up testing of foundry_cache.py * refining tests * refining tests * tighten up testing, remove downloads from GHA, hold off on https download until next issue * updating workflow * removing test for GHA * removing test for GHA * removing test for GHA * removing test for GHA * removing test for GHA * removing test for GHA * removing test for GHA * removing test for GHA * removing test for GHA * removing test for GHA * removing test for GHA * skipping problematic test in GHA * skipping problematic test in GHA * skipping problematic test in GHA * skipping problematic test in GHA * skipping problematic test in GHA * skipping problematic test in GHA * skipping problematic test in GHA * skipping problematic test in GHA * skipping problematic test in GHA * testing bump of pytest version * reverting to skip test that uses pytest.raises in GHA * addressing feedback from PR --------- Co-authored-by: KJ <kj.schmidt913@gmail.com> Co-authored-by: ascourtas <ascourtas@gmail.com> Co-authored-by: repo-visualizer <repo-visualizer@users.noreply.github.com> Co-authored-by: Ben Blaiszik <blaiszik@uchicago.edu> Co-authored-by: BraedenCu <braedencullen120@gmail.com> Co-authored-by: Aadit-Ambadkar <aadit.ambadkar@gmail.com> Co-authored-by: Aadit Ambadkar <58674441+Aadit-Ambadkar@users.noreply.github.com> Co-authored-by: Isaac Darling <61167692+isaac-darling@users.noreply.github.com> Co-authored-by: Logan Ward <WardLT@users.noreply.github.com> Co-authored-by: C. Y. Schneck <22159116+cyschneck@users.noreply.github.com> Co-authored-by: Sterling G. Baird <45469701+sgbaird@users.noreply.github.com> Co-authored-by: Logan Ward <ward.logan.t@gmail.com> Co-authored-by: Eric Blau <blau@anl.gov> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Marshall McDonnell <mcdonnellmt@ornl.gov>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR addresses #189 by creating a github action that automatically builds API documentation from docstrings. The action is set up to be triggered by a successful PR merge, and then generates the .md files into the
docs/folder of the PR base branch. The action employs the lazydocs package as after extensive exploration, it was the simplest and cleanest solution that I found to generate vanilla .md files. Note that there was no clear way to generate API documentation from the docstrings that integrated with gitbook - as such, the .md files created are not currently integrated into the gitbook. This documentation would also benefit from additional development to refine the docstrings.