-
Notifications
You must be signed in to change notification settings - Fork 21.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add #download_with_index and #download_chunk methods to blob instances #41297
Add #download_with_index and #download_chunk methods to blob instances #41297
Conversation
If you need to download files by chunks, you could send a block to the download method. However, if for some reason the execution of this block failed, there was no way to resume the processing from the last successfully processed chunk. To take advantage of those previous chunks, it was necessary to re-implement the reading of files by chunks, using the `download_chunk` method, keeping a record of the execution point and updating offsets (something like a local pagination of chunks) With the `download_with_index` method we will receive the chunk and (optionally) the current index in each block. In case of failures, you only need to restart the execution with the index ```ruby a_tb_file.download_with_index(last_successfully_processed_chunk_index) do |chunk, index| # ... last_successfully_processed_chunk_index = index end ```
If you need to download files by chunks, you could send a block to the download method. However, if for some reason the execution of this block failed, there was no way to resume the processing from the last successfully processed chunk. To take advantage of those previous chunks, it was necessary to re-implement the reading of files by chunks, using the `download_chunk` method, keeping a record of the execution point and updating offsets (something like a local pagination of chunks) With the `download_with_index` method we will receive the chunk and (optionally) the current index in each block. In case of failures, you only need to restart the execution with the index ```ruby a_tb_file.download_with_index(last_successfully_processed_chunk_index) do |chunk, index| # ... last_successfully_processed_chunk_index = index end ``` Fix #download_chunk for blob instances was missing
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
🦄 |
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
Add #download_with_index and #download_chunk methods to blob instances |
Add #download_with_index and #download_chunk methods to blob instances
If you need to download files by chunks, you could send a block to the download method.
However, if for some reason the execution of this block failed, there was no way to resume the processing from the last successfully processed chunk.
To take advantage of those previous chunks, it was necessary to re-implement the reading of files by chunks, using the
download_chunk
method, keeping a record of the execution point and updating offsets (something like a local pagination of chunks)With the
download_with_index
method we will receive the chunk and (optionally) the current index in each block.In case of failures, you only need to restart the execution with the index