Refresh stage metadata automatically in the background to avoid exceptions encountered during file upload in uploadWithoutConnection

[ademfreshbooks]

Snowflake Ingest SDK: 2.1.0

For long running clients in streaming mode, It seems ~ every hour, session expires for StreamingIngestStage (used by FlushService), that raises:

Exception encountered during file upload in uploadWithoutConnection

Then StreamingIngestStage retries request again, by default 5 times.

Usually this does the job, however for app observability sake, this is not perfect as this creates false positives.

Also, there does not appear to be any configuration options available to avoid this behaviour, or options to override some of the things.

Can you please try to add auto refresh for stage metadata to avoid this situation?

[sfc-gh-lsembera]

Thanks for the suggestion, @ademfreshbooks. We do attempt to auto-refresh stage metadata when uploads start failing, but we can improve the situation by not logging the initial exception when metadata was immediately successfully refreshed.

[ademfreshbooks]

Thanks @sfc-gh-lsembera

Upon deeper inspection, it seems the error is being logged in net.snowflake.client.jdbc.SnowflakeFileTransferAgent here before it is being thrown to upper layer.

So it seems Snowflake Ingest SDK is already doing what you say, ie not logging first exception as error as shown here.

What would be your recommendation how to address this behaviour?

It would be ideal to have flag to disable this particular log call...

[sfc-gh-lsembera]

Yeah, the SDK doesn't log the first upload error, but the library we are using to upload data does. Thanks for reporting the issue, we will post updates here once the issue gets prioritized.

[sfc-gh-tzhang]

This should be fixed in 2.1.2 release, please give it a try and report back if needed, thanks!