[FEATURE] Allowing schema to be passed in as batch_spec_passthrough in Spark
#5900
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for niobium-lead-7998 ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
👇 Click on the image for a new way to code review
Legend |
Shinnnyshinshin
commented
Aug 31, 2022
Comment on lines
+281
to
+282
| @pre_dump | ||
| def prepare_dump(self, data, **kwargs): |
There was a problem hiding this comment.
batch_spec_passthrough can be at the Asset-level
Shinnnyshinshin
commented
Aug 31, 2022
Comment on lines
+764
to
+765
| @pre_dump | ||
| def prepare_dump(self, data, **kwargs): |
There was a problem hiding this comment.
batch_spec_passthrough can be at the DataConnector-level
Shinnnyshinshin
force-pushed
the
f/GREAT-465/GREAT-1204/adding-serialization-for-schema-in-spark
branch
from
67a0f5e
to
22f5c01
Compare
…on-for-schema-in-spark
Shinnnyshinshin
changed the title
schema in Sparkschema to be passed in as batch_spec_passthrough in Spark
Shinnnyshinshin
commented
Sep 1, 2022
Comment on lines
+713
to
+716
| @pytest.mark.parametrize( | ||
| "checkpoint_config,expected_serialized_checkpoint_config", | ||
| [ | ||
| pytest.param( |
There was a problem hiding this comment.
the same tests are now parameterized :)
Shinnnyshinshin
commented
Sep 1, 2022
Comment on lines
-772
to
-778
| @pytest.mark.integration | ||
| def test_serialization_of_datasource_with_nested_objects_spark(spark_session): | ||
| datasource_config: DatasourceConfig = DatasourceConfig( | ||
| name="taxi_data", | ||
| class_name="Datasource", | ||
| module_name="great_expectations.datasource", | ||
| execution_engine=ExecutionEngineConfig( |
There was a problem hiding this comment.
the same tests are now parameterized :)
added 3 commits
August 31, 2022 17:31
…in-spark' of https://github.com/great-expectations/great_expectations into f/GREAT-465/GREAT-1204/adding-serialization-for-schema-in-spark * 'f/GREAT-465/GREAT-1204/adding-serialization-for-schema-in-spark' of https://github.com/great-expectations/great_expectations: [BUGFIX] Patch issue with `checkpoint_identifier` within `Checkpoint.run` workflow (#5894) [MAINTENANCE] Adding serialization tests for Spark (#5897) [MAINTENANCE] Add slow pytest marker to config and sort them alphabetically. (#5892)
This reverts commit 41dd255.
NathanFarmer
approved these changes
Sep 1, 2022
| if schema and isinstance(schema, StructType): | ||
| data["batch_spec_passthrough"]["reader_options"][ | ||
| "schema" | ||
| ] = schema.jsonValue() |
tests/core/test_serialization.py
Outdated
Comment on lines
945
to
953
| "schema": StructType( | ||
| [ | ||
| StructField( | ||
| "a", IntegerType(), True, None | ||
| ), | ||
| StructField( | ||
| "b", IntegerType(), True, None | ||
| ), | ||
| ] |
There was a problem hiding this comment.
This is leveling up huge!!
added 2 commits
August 31, 2022 20:04
…on-for-schema-in-spark
added 2 commits
September 1, 2022 08:43
…on-for-schema-in-spark
…on-for-schema-in-spark * develop: [MAINTENANCE] Remove xfails from passing tests in preparation for 0.15.21 release (#5908)
added 2 commits
September 1, 2022 10:50
…on-for-schema-in-spark * develop: [DOCS] DOC-368 spelling correction (#5912)
6 tasks
…on-for-schema-in-spark
Shinnnyshinshin
deleted the
f/GREAT-465/GREAT-1204/adding-serialization-for-schema-in-spark
branch
6 tasks
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes proposed in this pull request:
StructType, which is not serializable.schema.jsonValue()call which translates the object into ajson, whenever a schema is passed in as abatch_spec_passthroughparameterschemacan be passed in at theAsset-level andDataConnector-level.schemacan also be passed in as part ofBatchRequestthat is part of aCheckpoint.prepare_dumpandconvert_to_json_serializable()and adds integration tests totest_serialization.py.SparkDFExecutionEngineable to useschema#5917Definition of Done
Please delete options that are not relevant.
Thank you for submitting!