-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEATURE] DataAsset uses partitioner from BatchConfig #9499
Merged
joshua-stauffer
merged 75 commits into
develop
from
f/v1-175/asset_uses_partitioner_from_batch_request
Feb 26, 2024
Merged
Changes from all commits
Commits
Show all changes
75 commits
Select commit
Hold shift + click to select a range
c3969dd
wip
joshua-stauffer c9d892a
Merge branch 'develop' into f/v1-175/generic_partitioners
joshua-stauffer 3cc0639
add generic partitioners
joshua-stauffer 36239d9
move Partitioner out of typechecking
joshua-stauffer 4854727
update import
joshua-stauffer 19164a2
type
joshua-stauffer 8bf119b
schema update
joshua-stauffer e695d19
add partitioner to BatchRequest
joshua-stauffer a167db1
update tests
joshua-stauffer 50994e7
update test snapshot
joshua-stauffer e20b62e
add partitioner to DataAsset.build_batch_request
joshua-stauffer 307263b
update type stub
joshua-stauffer 5032352
update test snapshot
joshua-stauffer 8193139
missed one
joshua-stauffer fa5cda8
Merge branch 'develop' into f/v1-175/add_partitioners_to_batch_request
joshua-stauffer 9218fc9
add test
joshua-stauffer f8780c2
add partitioner map to base class
joshua-stauffer c7e9103
add partitioner maps
joshua-stauffer 29e28af
rename spark partitioners
joshua-stauffer c0d2c1a
refactor sql partitioners
joshua-stauffer af5e0d9
rename spark partitioners
joshua-stauffer 8795e5f
Merge branch 'develop' into f/v1-175/add_partitioners_to_batch_request
joshua-stauffer 9d2bdf5
schema update
joshua-stauffer f9b2210
Merge branch 'f/v1-175/add_partitioners_to_batch_request' into f/v1-1…
joshua-stauffer 89f7193
update per rename
joshua-stauffer 4b405a0
Merge branch 'develop' into f/v1-175/asset_uses_partitioner_from_batc…
joshua-stauffer 7bb9777
implement method to replace batch_options property
joshua-stauffer 1c0c5d7
move partitioner resolve method to subclass
joshua-stauffer baff01e
typeguard
joshua-stauffer 364dfaa
add override
joshua-stauffer f0d2f52
rename param
joshua-stauffer d626182
update types
joshua-stauffer a4e24b4
remove from interface
joshua-stauffer 269089e
add types to subclass
joshua-stauffer f00a1fb
tweak type
joshua-stauffer dbd95d0
revert
joshua-stauffer 1004b30
types
joshua-stauffer 8623707
move type out of typechecking
joshua-stauffer ac3ee51
move import
joshua-stauffer 85f072a
try type as function
joshua-stauffer 9ed5cee
revert
joshua-stauffer c2ba4c7
hacky type workaround
joshua-stauffer fbf7657
use Type
joshua-stauffer abc76d5
spark: use batch request partitioner
joshua-stauffer 0cf9e3a
sql: use batch request partitioner
joshua-stauffer de6fc0c
add sqlite partitioners
joshua-stauffer 454aef8
fix sqlite tests
joshua-stauffer 7a9fafa
use get_batch_request_options instead of batch_request_options property
joshua-stauffer 6edf63f
wip
joshua-stauffer 7da1552
rename method
joshua-stauffer aa244ab
sqlite tests passing
joshua-stauffer d0b3961
update signature
joshua-stauffer ab1149f
update types
joshua-stauffer e2e4d16
move ConvertedDatetime partitioner into sql
joshua-stauffer d801d17
ensure sqlite assets have access to correct partitioner map
joshua-stauffer f97c441
schema update
joshua-stauffer d88a277
Merge branch 'develop' into f/v1-175/asset_uses_partitioner_from_batc…
joshua-stauffer d0ff1ca
schema update
joshua-stauffer b924f97
refactor partitioner to batch request in conftest
joshua-stauffer 7b7247f
update integration tests to use partitioner in batch request
joshua-stauffer 3362915
remove deprecated partitioner
joshua-stauffer 7611ecb
Merge branch 'develop' into f/v1-175/asset_uses_partitioner_from_batc…
joshua-stauffer ff5abdf
schema update
joshua-stauffer 6137d92
update integration tests
joshua-stauffer 1ae3497
update viral snippet
joshua-stauffer d91e36d
update tests related to fixture
joshua-stauffer 6c964ad
update postgresql tests
joshua-stauffer e7d28e3
update snippets
joshua-stauffer 175814b
public api
joshua-stauffer 53d8c89
schema update
joshua-stauffer 4d3b6ee
update api excludes
joshua-stauffer 4eab300
fix doc snippet
joshua-stauffer 4b0651e
remove assert
joshua-stauffer 43888ed
remove comment
joshua-stauffer 9d42d57
remove batch config option keys method from batch config
joshua-stauffer File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we do any validation on this string?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
current behavior is to not validate the string, so i think it's likely okay to not. I'm not 100% how we would validate it, either, since its correctness depends on the shape of the source data.