Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[MAINTENANCE] Test definition for ExpectColumnValuesToBeJsonParsable, including a fix for Spark #6207

Merged
merged 11 commits into from
Oct 27, 2022
Merged

[MAINTENANCE] Test definition for ExpectColumnValuesToBeJsonParsable, including a fix for Spark #6207

merged 11 commits into from
Oct 27, 2022

Conversation

Shinnnyshinshin
Copy link
Contributor

@Shinnnyshinshin Shinnnyshinshin commented Oct 26, 2022 •
edited
Loading

Changes proposed in this pull request:

  • Adds json-defined test for ExpectColumnValuesToBeJsonParsable
  • Fixes bug for column_values_json_parsable Metric, which erroneously asked for json_schema to be provided.
  • Closes DX-19

Definition of Done

  • My code follows the Great Expectations style guide
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added unit tests where applicable and made sure that new and existing tests are passing.
  • I have run any local integration tests and made sure that nothing is broken.

@netlify
Copy link

netlify bot commented Oct 26, 2022 •
edited
Loading

✅ Deploy Preview for niobium-lead-7998 ready!

Name Link
🔨 Latest commit c44093c
🔍 Latest deploy log https://app.netlify.com/sites/niobium-lead-7998/deploys/6359c8ae6e43b00009ed0b5b
😎 Deploy Preview https://deploy-preview-6207--niobium-lead-7998.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

Shinnnyshinshin
Shinnnyshinshin commented Oct 26, 2022
View reviewed changes
great_expectations/expectations/metrics/column_map_metrics/column_values_json_parseable.py
Comment on lines 28 to +29
@column_condition_partial(engine=SparkDFExecutionEngine)
def _spark(cls, column, json_schema, **kwargs):
def _spark(cls, column, **kwargs):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was the bug. json_schema is not required by the ColumnValuesJsonParseable metric, but was required by the _spark partial function.

@ghost
Copy link

ghost commented Oct 26, 2022 •
edited by ghost
Loading

👇 Click on the image for a new way to code review

  • Make big changes easier — review code in small groups of related files

  • Know where to start — see the whole change at a glance

  • Take a code tour — explore the change with an interactive tour

  • Make comments and review — all fully sync’ed with github

    Try it now!

Review these changes using an interactive CodeSee Map

Legend

CodeSee Map Legend

@Shinnnyshinshin Shinnnyshinshin self-assigned this Oct 26, 2022
Will Shin added 4 commits October 25, 2022 18:00
updating number
28590fa
Merge branch 'develop' into m/DX-19/adding-json-sample-data-expect-co…
854f40c 
…lumn-values-json-parseable

* develop:
  [MAINTENANCE] Replace deprecated `iteritems` with `items` (#6205)
  [MAINTENANCE] Add validate_configuration method to expect_table_row_count_to_equal_other_table (#6209)
removing flags that have been added too early
c9be833
adding last test to gallery
359653d
@Shinnnyshinshin Shinnnyshinshin marked this pull request as ready for review October 26, 2022 16:13
kenwade4
kenwade4 approved these changes Oct 26, 2022
View reviewed changes
Copy link
Contributor

@kenwade4 kenwade4 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running build_gallery local has good output as well!

(run_diagnostics) coverage_score: 14.0 for expect_column_values_to_be_json_parseable ... engines: 2, backends: 2, passing tests: 10, failing tests:0

@Shinnnyshinshin Shinnnyshinshin enabled auto-merge (squash) October 26, 2022 22:02
@Shinnnyshinshin Shinnnyshinshin merged commit 2fbb902 into develop Oct 27, 2022
@Shinnnyshinshin Shinnnyshinshin deleted the m/DX-19/adding-json-sample-data-expect-column-values-json-parseable branch October 27, 2022 01:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kenwade4 kenwade4 kenwade4 approved these changes

@anthonyburdi anthonyburdi Awaiting requested review from anthonyburdi

Assignees

@Shinnnyshinshin Shinnnyshinshin

Labels
core-team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Shinnnyshinshin @kenwade4