Skip to content

Conversation

@ebyhr
Copy link
Contributor

@ebyhr ebyhr commented Aug 10, 2025

Iceberg throws exceptions if JSON contains default values on timestamp nanos:

Caused by: java.lang.ArithmeticException: long overflow
	at java.base/java.lang.Math.multiplyExact(Math.java:1035)
	at org.apache.iceberg.util.DateTimeUtil.microsToNanos(DateTimeUtil.java:101)
	at org.apache.iceberg.expressions.Literals$TimestampLiteral.to(Literals.java:447)
	at org.apache.iceberg.expressions.Literals$LongLiteral.to(Literals.java:306)
	at org.apache.iceberg.types.Types$NestedField.castDefault(Types.java:890)
	at org.apache.iceberg.types.Types$NestedField.<init>(Types.java:882)
	at org.apache.iceberg.types.Types$NestedField$Builder.build(Types.java:850)
	at org.apache.iceberg.SchemaParser.structFromJson(SchemaParser.java:242)
	at org.apache.iceberg.SchemaParser.typeFromJson(SchemaParser.java:186)
	at org.apache.iceberg.SchemaParser.fromJson(SchemaParser.java:277)
	at org.apache.iceberg.util.JsonUtil.parse(JsonUtil.java:104)
	at org.apache.iceberg.SchemaParser.lambda$fromJson$1(SchemaParser.java:296)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2704)
	at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1932)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2702)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2684)
	at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112)
	at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:63)
	at org.apache.iceberg.SchemaParser.fromJson(SchemaParser.java:296)

@github-actions github-actions bot added the core label Aug 10, 2025
@ebyhr ebyhr marked this pull request as draft August 10, 2025 22:11
@ebyhr ebyhr force-pushed the ebi/timestamp-nanos-schema-parsers branch from cde870a to 4c8c522 Compare August 10, 2025 22:43

private static Literal<?> defaultFromJson(String defaultField, Type type, JsonNode json) {
if (json.has(defaultField)) {
return Expressions.lit(SingleValueParser.fromJson(type, json.get(defaultField)));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a problem for 1.10? Should we block the release on this? While I know the chances are unlikely here but if anyone is trying to set defaults close to Epoch they won't Overflow and will instead write incorrect values

return Expressions.lit(SingleValueParser.fromJson(type, json.get(defaultField)));
Object value = SingleValueParser.fromJson(type, json.get(defaultField));
if (type instanceof Types.TimestampNanoType) {
return Expressions.nanos((long) value);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we check value not null? the json value itself can also be null, right?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could precondition this, we will have slightly divergent behavior though. The normal path would bail out here

https://github.com/apache/iceberg/blob/d5476cae1b4e9b560f0126b7b9117630da9a37ca/api/src/main/java/org/apache/iceberg/expressions/Literals.java#L58C6-L61C101

So as written this would throw an NPE at a different spot only for nanos

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, this is not an issue. the spec says

Non-null values for initial-default or write-default are invalid.

Earlier, I wasn't sure if null values are allowed for the default attributes.

@github-actions
Copy link

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions bot added the stale label Sep 15, 2025
@github-actions
Copy link

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions bot closed this Sep 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants