Building or uploading blob failed

[charlie-firth-cko]

Hey SF - I was hoping I could get some help.

I'm encountering this weird error when using SNOWPIPE_STREAMING & Schematization and I'm not sure how to debug further - I can't see any errors in Snowflake and the error message here isn't too clear.

Is there some sort of HISTORY view in Snowflake that I can use to get further details on failures when using SNOWPIPE STREAMING, or do I need to jump into the code to get better logging?

This is on version 1.9.1 (though I have made some changes locally)

Example of th error:

[2023-03-09 12:20:57,750] ERROR [card-charge-nested-snowflake-sink|task-2] [SF_INGEST] Building or uploading blob failed, client=KC_CLIENT_card_charge_nested_snowflake_sink_336255224_2, file=2023/3/9/12/20/rr96ax_eFZipMNBB7d0je1zwuBUFHvLQmhEbpov3gtY5pItd4GQCC_1011_4804_127.bdec, exception=java.util.concurrent.ExecutionException: net.snowflake.ingest.utils.SFException: Ingest client internal error: Ingest client internal error: Null count bigger than total row count on col=EVENT_METADATA_ITEMS, nullCount=64, rowCount=63.., detail=net.snowflake.ingest.utils.SFException: Ingest client internal error: Ingest client internal error: Null count bigger than total row count on col=EVENT_METADATA_ITEMS, nullCount=64, rowCount=63.., cause=net.snowflake.ingest.utils.SFException: Ingest client internal error: Ingest client internal error: Null count bigger than total row count on col=EVENT_METADATA_ITEMS, nullCount=64, rowCount=63.., cause_detail=Ingest client internal error: Ingest client internal error: Null count bigger than total row count on col=EVENT_METADATA_ITEMS, nullCount=64, rowCount=63.., cause_trace=

Thanks!

[sfc-gh-japatel]

Do you know which ingest SDK you are using in your lib folder?

[sfc-gh-tzhang]

@charlie-firth-cko we're not able to repro this issue, would you be able to share an repro with us? Thanks!

[charlie-firth-cko]

Sorry for the delay, went on holiday!

We're using version 1.1.0 of the Ingest SDK.

I've made some changes locally to our version of the connector which might be causing this, but I was hoping there was a way to debug this a bit better on my side. Is there was some additional logging I could add, or a way to access the errors from Snowflake?

Based on the error message referencing a col Null count bigger than total row count on col=EVENT_DATA_METADATA_WALLET_PROVIDER i'm assuming there's a certain data structure that's causing this - unfortunately the topic in question ingests around 500 rec/sec so it's quite hard to nail down which message it is to repro.

Here are some of the surrounding logs:

[2023-03-09 15:30:39,594] ERROR [card-charge-nested-snowflake-sink\|task-0] [SF_INGEST] Building blob failed, client=KC_CLIENT_card_charge_nested_snowflake_sink_336255224_0, file=2023/3/9/15/30/rr9f33_qM3rJ9mBBfKzvk3IqMwRKFBwFgaZX5ONUQ1YXBB2Hc2fACC_1011_265_135.bdec, exception=net.snowflake.ingest.utils.SFException: Ingest client internal error: Null count bigger than total row count on col=EVENT_DATA_METADATA_WALLET_PROVIDER, nullCount=53, rowCount=51., detail=Ingest client internal error: Null count bigger than total row count on col=EVENT_DATA_METADATA_WALLET_PROVIDER, nullCount=53, rowCount=51., trace=
--
net.snowflake.ingest.streaming.internal.EpInfo.verifyEpInfo(EpInfo.java:33)
net.snowflake.ingest.streaming.internal.AbstractRowBuffer.buildEpInfoFromStats(AbstractRowBuffer.java:544)
net.snowflake.ingest.streaming.internal.BlobBuilder.constructBlobAndMetadata(BlobBuilder.java:130)
net.snowflake.ingest.streaming.internal.FlushService.buildAndUpload(FlushService.java:450)
net.snowflake.ingest.streaming.internal.FlushService.lambda$distributeFlushTasks$9(FlushService.java:386)
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base/java.lang.Thread.run(Thread.java:829), all channels in the blob will be invalidated (net.snowflake.ingest.streaming.internal.FlushService:51)
[2023-03-09 15:30:39,595] ERROR [card-charge-nested-snowflake-sink\|task-0] [SF_INGEST] Building or uploading blob failed, client=KC_CLIENT_card_charge_nested_snowflake_sink_336255224_0, file=2023/3/9/15/30/rr9f33_qM3rJ9mBBfKzvk3IqMwRKFBwFgaZX5ONUQ1YXBB2Hc2fACC_1011_265_135.bdec, exception=java.util.concurrent.ExecutionException: net.snowflake.ingest.utils.SFException: Ingest client internal error: Ingest client internal error: Null count bigger than total row count on col=EVENT_DATA_METADATA_WALLET_PROVIDER, nullCount=53, rowCount=51.., detail=net.snowflake.ingest.utils.SFException: Ingest client internal error: Ingest client internal error: Null count bigger than total row count on col=EVENT_DATA_METADATA_WALLET_PROVIDER, nullCount=53, rowCount=51.., cause=net.snowflake.ingest.utils.SFException: Ingest client internal error: Ingest client internal error: Null count bigger than total row count on col=EVENT_DATA_METADATA_WALLET_PROVIDER, nullCount=53, rowCount=51.., cause_detail=Ingest client internal error: Ingest client internal error: Null count bigger than total row count on col=EVENT_DATA_METADATA_WALLET_PROVIDER, nullCount=53, rowCount=51.., cause_trace=
net.snowflake.ingest.streaming.internal.FlushService.lambda$distributeFlushTasks$9(FlushService.java:418)
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base/java.lang.Thread.run(Thread.java:829) all channels in the blob will be invalidated (net.snowflake.ingest.streaming.internal.RegisterService:51)
[2023-03-09 15:30:39,596] WARN [card-charge-nested-snowflake-sink\|task-0] Unexpected error happens during insertRows: {} (net.snowflake.ingest.streaming.internal.AbstractRowBuffer:76)
java.lang.NullPointerException
[2023-03-09 15:30:39,596] INFO [card-charge-nested-snowflake-sink\|task-0] [SF_INGEST] Trying to close ParquetRowBuffer for channel=LANDING.DP.CARD_CHARGE_FLATTENED_NESTED.CARD-CHARGE-EVENT-STREAM-ALL-PROD_0 from function=invalidate, allocatedBeforeRelease=0, allocatedAfterRelease=0 (net.snowflake.ingest.streaming.internal.AbstractRowBuffer:58)
[2023-03-09 15:30:39,597] WARN [card-charge-nested-snowflake-sink\|task-0] Channel is invalidated, name=LANDING.DP.CARD_CHARGE_FLATTENED_NESTED.CARD-CHARGE-EVENT-STREAM-ALL-PROD_0, channel sequencer=44535, row sequencer=1, message=invalidate with matched sequencer (net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestChannelInternal:76)
[2023-03-09 15:30:39,597] INFO [card-charge-nested-snowflake-sink\|task-0] [SF_INGEST] Trying to close ParquetRowBuffer for channel=LANDING.DP.CARD_CHARGE_FLATTENED_NESTED.CARD-CHARGE-EVENT-STREAM-ALL-PROD_0 from function=checkValidation, allocatedBeforeRelease=0, allocatedAfterRelease=0 (net.snowflake.ingest.streaming.internal.AbstractRowBuffer:58)
[2023-03-09 15:30:39,597] WARN [card-charge-nested-snowflake-sink\|task-0] [SF_KAFKA_CONNECTOR] [KC:158131924] Failed Attempt to invoke the insertRows API for buffer: Buffer: StreamingBuffer{numOfRecords=1145, bufferSizeBytes=5002905, firstOffset=5205842007, lastOffset=5205843151}
LastExcep: net.snowflake.ingest.utils.SFException: Channel LANDING.DP.CARD_CHARGE_FLATTENED_NESTED.CARD-CHARGE-EVENT-STREAM-ALL-PROD_0 is invalid and might contain uncommitted rows, please consider reopening the channel to restart.
LastRes: {} (com.snowflake.kafka.connector.internal.streaming.TopicPartitionChannel:143)
[2023-03-09 15:30:39,598] WARN [card-charge-nested-snowflake-sink\|task-0] [SF_KAFKA_CONNECTOR] [KC:158131924] [INSERT_ROWS_FALLBACK] Re-opening channel:LANDING.DP.CARD_CHARGE_FLATTENED_NESTED.CARD-CHARGE-EVENT-STREAM-ALL-PROD_0 (com.snowflake.kafka.connector.internal.streaming.TopicPartitionChannel:143)
[2023-03-09 15:30:39,598] INFO [card-charge-nested-snowflake-sink\|task-0] [SF_KAFKA_CONNECTOR] [KC:158131924] Opening a channel with name:card-charge-event-stream-all-prod_0 for table name:CARD_CHARGE_FLATTENED_NESTED (com.snowflake.kafka.connector.internal.streaming.TopicPartitionChannel:107)

[sfc-gh-tzhang]

@charlie-firth-cko thanks, could you tell us more about the column EVENT_DATA_METADATA_WALLET_PROVIDER? What's the data type? Whether the value is mostly NULLs? Is it a column that exists since table creation or something you added later? Also it would be good if you could send us the full log file if possible.

[sfc-gh-tzhang]

@charlie-firth-cko actually @sfc-gh-lsembera just identified the issue and we will fix it in the next release. For workaround, please use OpenChannelRequest.OnErrorOption.ABORT instead of OpenChannelRequest.OnErrorOption.CONTINUE when creating a channel.

[charlie-firth-cko]

Amazing, thankyou! 🚀

[charlie-firth-cko]

Hey SF, as a quick update on this - using OnError.ABORT broke schema evolution, as the task would just error out instead of looking to evolve the tables schema.

Could you let me know what the root cause is?

[sfc-gh-tzhang]

Not sure if I understand, but KC should always use OnError=CONTINUE, otherwise throwing an exception is the expected behavior

[sfc-gh-tzhang]

Closing since this is fixed