Unable to download entire rds log file with download_db_log_file_portion

[wr0ngway]

I'm using aws-sdk 2.1.35 to try and download a postgresql rds log file

When using either the block based pagination, or doing it manually, I'm unable to download the entire rds log file. The entire log file is 260MB, but either method only gets 100MB. The final paged request shows additional_data_pending=false, marker ="17:105920837" for either method. I am able to download the entire file using the web console. The code I'm using is shown below.

Is there some other way to download the entire log file? That is, a way to get a handle on the signed url the console generates to download the entire thing directly would be great.

      manual_pagination = true
      open(path, 'wb+') do |f|

        if manual_pagination
          opts = {
            db_instance_identifier: db_id,
            log_file_name: file,
            marker: "0"
          }
          additional_data_pending = true
          while additional_data_pending do
            print "."
            out = rds.download_db_log_file_portion(opts)
            f.write(out[:log_file_data])
            opts[:marker] = out[:marker]
            additional_data_pending = out[:additional_data_pending]
          end
          puts "."
        else
          rds.download_db_log_file_portion(db_instance_identifier: db_id,
                                           log_file_name: file, marker: "0").each do |r|
            print "."
            f.write r[:log_file_data]
          end
          puts "."
        end

[wr0ngway]

If I manually start a new request with the marker returned from the last request, the download proceeds, but eventually fails again in a similar way. I am thus able to get all the way to the end of the file with some retry logic. I noticed that the size of the returned data is 1048576 bytes when additional_data_pending=false, but 1048611 when true, so I'm guessing something about the data or how it's processed is causing the abort.

[wr0ngway]

Here is a work around that works:

      open(path, 'wb+') do |f|

        opts = {
          db_instance_identifier: db_id,
          log_file_name: file,
          marker: "0"
        }

        while true do
          print "."
          out = rds.download_db_log_file_portion(opts)
          f.write(out[:log_file_data])

          break if opts[:marker] == out[:marker]
          opts[:marker] = out[:marker]
        end
        puts "."

      end

[awood45]

That makes sense, this looks essentially like multipart download functionality.

Looking at this as a feature request now.

[awood45]

Closing this issue, I've added the feature request and it looks like we have a workaround in the meantime (the feature would just be a high level wrapper around this same workaround).

Feel free to reach out if you have any questions or further issues.