refactor: change PartitionSpec to be independent of Schema #299
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
HeartLinked
force-pushed
the
partition_spec_1
branch
from
335a0da to
bb499ad
Compare
wgtmac
reviewed
Nov 7, 2025
wgtmac
requested changes
Nov 8, 2025
Member
wgtmac
left a comment
wgtmac
left a comment
There was a problem hiding this comment.
My main feedback is that we need to keep the schema parameter because we eventually need it to create bound partition spec.
| /// \param[out] default_spec_id The default partition spec ID. | ||
| /// \param[out] partition_specs The list of partition specs. | ||
| Status ParsePartitionSpecs(const nlohmann::json& json, int8_t format_version, | ||
| const std::shared_ptr<Schema>& current_schema, |
src/iceberg/json_internal.cc
Outdated
| for (const auto& spec_json : spec_array) { | ||
| ICEBERG_ASSIGN_OR_RAISE(auto spec, | ||
| PartitionSpecFromJson(current_schema, spec_json)); | ||
| ICEBERG_ASSIGN_OR_RAISE(auto spec, PartitionSpecFromJson(spec_json)); |
lishuxu
reviewed
Nov 8, 2025
| std::shared_ptr<Schema> schema) { | ||
| if (fields_.empty()) { | ||
| return nullptr; | ||
| } |
Contributor
There was a problem hiding this comment.
The current implementation allows using partition_type_ cache based on tableschema_, but the given schema parameter may belong to a different schema version, which could lead to correctness issues.
Suggest adding schema_id to the cache key to ensure consistency.
such as:
std::unordered_map<int32_t, std::shared_ptr> partition_type_cache_;
wgtmac
requested changes
Nov 10, 2025
HeartLinked
force-pushed
the
partition_spec_1
branch
from
a859eac to
f63808c
Compare
HeartLinked
force-pushed
the
partition_spec_1
branch
from
f63808c to
8455f61
Compare
wgtmac
reviewed
Nov 10, 2025
wgtmac
changed the title
wgtmac
approved these changes
Nov 10, 2025
wgtmac
reviewed
Nov 10, 2025
HeartLinked
force-pushed
the
partition_spec_1
branch
from
baafb89 to
eb0e134
Compare
wgtmac
requested changes
Nov 11, 2025
wgtmac
approved these changes
Nov 11, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
schema_member fromPartitionSpec, MakingPartitionType()method require a Schema parameter instead of using the stored schema.table_schema_inManifestEntryAdapter.table_schemainmetadata_["schema"].