Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] FDS Deployment Pattern - AWS S3 Pandas #7718

Merged
merged 45 commits into from
Apr 27, 2023

Conversation

Shinnnyshinshin
Copy link
Contributor

@Shinnnyshinshin Shinnnyshinshin commented Apr 25, 2023

Changes proposed in this pull request:

  • Updated How to use GX with AWS using Cloud Storage and Pandas doc to refer to script.
  • Updated components that are used by the doc. Was not able to remove all components because they were referred to in other Docs versions.
  • Added new integration test: tests/integration/docusaurus/deployment_patterns/aws_cloud_storage_pandas.py that tests the workflow.
  • Added test to test_script_runner.py

Definition of Done

  • My code follows the Great Expectations style guide
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added unit tests where applicable and made sure that new and existing tests are passing.
  • I have run any local integration tests and made sure that nothing is broken.

@netlify
Copy link

netlify bot commented Apr 25, 2023

Deploy Preview for niobium-lead-7998 canceled.

Name Link
🔨 Latest commit 29eac08
🔍 Latest deploy log https://app.netlify.com/sites/niobium-lead-7998/deploys/6449d4555add4c000854cdf4

@Shinnnyshinshin Shinnnyshinshin marked this pull request as draft April 25, 2023 00:28
@Shinnnyshinshin Shinnnyshinshin self-assigned this Apr 25, 2023
@ghost
Copy link

ghost commented Apr 25, 2023

👇 Click on the image for a new way to code review

Review these changes using an interactive CodeSee Map

Legend

CodeSee Map legend

<PythonTabTest />

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes are the result of the linter

@Shinnnyshinshin Shinnnyshinshin changed the title [DOCS] WIP - Deployment Pattern S3 Pandas [DOCS] FDS Deployment Pattern - AWS S3 Pandas Apr 25, 2023
@Shinnnyshinshin Shinnnyshinshin marked this pull request as ready for review April 25, 2023 16:54
@@ -1863,7 +1863,7 @@ def main():
# any methods or classes you are adding to documentation with the @public_api
# decorator and any relevant "new" or "deprecated" public api decorators.
# If the actual is lower than the threshold, please reduce the threshold.
PUBLIC_API_MISSING_THRESHOLD = 92 # TODO: reduce this number again once this works for the Fluent DS dynamic methods
PUBLIC_API_MISSING_THRESHOLD = 99 # TODO: reduce this number again once this works for the Fluent DS dynamic methods
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This had to be raised because I refer to more FDS methods in my script

…rage_pandas.py

Co-authored-by: Gabriel <gabriel59kg@gmail.com>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

Looking at the deploy preview here I think most of these extra imports are unnecessary.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was as miss on my part. The section was referring to a different (spark s3 data context) doc. Doc has been updated.

Copy link
Member

@Kilo59 Kilo59 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found a small issue with some seemingly unnecessary imports but other than that it looks good to me.

It might be worth adding a note or something that users can for the most part complete Part 2 "connecting to data" without completing all of Part 1.

@Shinnnyshinshin Shinnnyshinshin enabled auto-merge (squash) April 26, 2023 18:47
@Shinnnyshinshin Shinnnyshinshin merged commit 627e119 into develop Apr 27, 2023
@Shinnnyshinshin Shinnnyshinshin deleted the m/_/deployment-pattern-s3-pandas branch April 27, 2023 02:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants