-
Notifications
You must be signed in to change notification settings - Fork 47
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stream can hang if there is an error reading a source file from S3 #414
Comments
Thank you for using this library, it would be great if you create PR so we can merge this change to mainline. Or i can revisit it little bit later, this problem worth to fix, thank you again for creating the issue this makes this library better. |
I've experimented with the v3 API and the problem persists. The workaround for v3 is: fs2.io.readInputStream(
blocker.blockOn(IO(
S3Client.create.getObject(
GetObjectRequest
.builder()
.bucket(bucket)
.key(key)
.build()
)
)),
chunkSize = 4096,
closeAfterUse = true,
blocker = blocker
) |
As far as I can tell, it seems this issue has been resolved. Tested today, and discovered through testing that running
|
Problem
We have an fs2 job that reads a JSON file from S3 and then processes this file with 30 parallel streams to extract different sets of data to write to 30 parquet files. We find that if there is an error reading the input file, such as the file key being incorrect or the S3 permissions being incorrect, then the job will hang, print no errors, and produce no output.
However, we also find that if we configure the job to produce fewer out files (which results in fewer parallel streams), then the job doesn't hang and an appropriate exception is produce and shows up in the logs.
Work Around
The problem appears to be that when the stream from S3 is read it does not use the
Blocker
that it should. To work around this we no longer usereadS3File
, but instead use our own version, which is as follows:The first difference from the fs2-aws API is that this
readFromS3
method takes aBlocker
instead of anExecutionContext
, which seems more correct as we already have aBlocker
in context to use for this. The second different from the fs2-aws API is thatblocker.blockOn(...)
is used to wrap theIO[InputSTream]
, which is what ultimately appears to fix the hanging problem.The text was updated successfully, but these errors were encountered: