Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds Pandas DF JSON materializer #347

Merged
merged 2 commits into from
Sep 15, 2023

Conversation

bryangalindo
Copy link
Contributor

This change adds the PandasJsonWriter to_json function to the existing Pandas materializer example script and updates the materialization notebook.

Changes

  • examples/pandas/materialization/notebook.ipynb
  • examples/pandas/materialization/my_script.py

How I tested this

  • Successfully ran unit tests for py3.11 locally using the following command in the root directory: pytest tests/
  • Successfully ran unit tests for py3.7-3.10 using docker run command found here.

Notes

N/A

Checklist

  • PR has an informative and human-readable title (this will be pulled into the release notes)
  • Changes are limited to a single goal (no scope creep)
  • Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future TODOs are captured in comments
  • Project documentation has been updated if adding/changing functionality.

@bryangalindo bryangalindo changed the title Adds Pandas JSON materializer Adds Pandas DF JSON materializer Sep 14, 2023
@bryangalindo bryangalindo marked this pull request as ready for review September 14, 2023 21:03
Copy link
Collaborator

@skrawcz skrawcz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The image in the notebook doesn't reflect what should be visualized. Otherwise the code looks great, thanks!

@skrawcz skrawcz mentioned this pull request Sep 15, 2023
@bryangalindo
Copy link
Contributor Author

The image in the notebook doesn't reflect what should be visualized. Otherwise the code looks great, thanks!

What needs to be corrected about the image below?

  • a) Should there be two separate pictures, one for each materializer?
  • b) Should there be one single picture, but two sets of independent DAG graphs for each materializer?
  • c) or something else?
image

@skrawcz
Copy link
Collaborator

skrawcz commented Sep 15, 2023

The image in the notebook doesn't reflect what should be visualized. Otherwise the code looks great, thanks!

What needs to be corrected about the image below?

  • a) Should there be two separate pictures, one for each materializer?
  • b) Should there be one single picture, but two sets of independent DAG graphs for each materializer?
  • c) or something else?
image

This image looks correct! I was meaning in the notebook the image isn't this one.

The one in the notebook that's checked in looks like:

Screen Shot 2023-09-14 at 10 27 32 PM

@bryangalindo
Copy link
Contributor Author

bryangalindo commented Sep 15, 2023

The image in the notebook doesn't reflect what should be visualized. Otherwise the code looks great, thanks!

What needs to be corrected about the image below?

  • a) Should there be two separate pictures, one for each materializer?
  • b) Should there be one single picture, but two sets of independent DAG graphs for each materializer?
  • c) or something else?
image

This image looks correct! I was meaning in the notebook the image isn't this one.

The one in the notebook that's checked in looks like:

Screen Shot 2023-09-14 at 10 27 32 PM

@skrawcz I think this was an issue with my PyCharm's Jupyter Notebook integration. I updated PyCharm and ran all the cells, so the checked-in notebook visualization should be updated.

Copy link
Collaborator

@skrawcz skrawcz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@skrawcz skrawcz merged commit abd9644 into DAGWorks-Inc:main Sep 15, 2023
@bryangalindo bryangalindo deleted the adds-pandas-json-materializer branch September 16, 2023 03:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants