Handling unprocessed_items and unprocessed_keys

[pk-n]

how are unprocessed_items for batch_write and unprocessed_keys for batch_read handled?

[pk-n]

I came up with a very simple approach for batch_write

      def batch_write_item table_name, objects, options = {}
        request_items = []
        options ||= {}
        objects.each do |o|
          request_items << { "put_request" => { item: o } }
        end

        begin
          resp = client.batch_write_item(
            {
              request_items: {
                table_name => request_items,
              },
              return_consumed_capacity: "TOTAL",
              return_item_collection_metrics: "SIZE"
            }.merge!(options)
          )
          unless resp.unprocessed_items.empty?
            sleep(2)
            unprocessed_objects = []
            resp.unprocessed_items[table_name].each do |request_item|
              unprocessed_objects << request_item.put_request.item
            end
            batch_write_item(table_name, unprocessed_objects, options)
          end
        rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException => e
          raise Dynamoid::Errors::ConditionalCheckFailedException, e
        end
      end

[pk-n]

I think I can also do the same check where I make a request(outside the gem) to batch_write_item.

[andrykonchin]

From the documentation:

If one or more of the following is true, DynamoDB rejects the entire batch write operation:

1. One or more tables specified in the BatchWriteItem request does not exist.

2. Primary key attributes specified on an item in the request do not match those in the corresponding table's primary key schema.

3. You try to perform multiple operations on the same item in the same BatchWriteItem request. For example, you cannot put and delete the same item in the same BatchWriteItem request.

4. There are more than 25 requests in the batch.

5. Any individual item in a batch exceeds 400 KB.

6. The total request size exceeds 16 MB.

https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html

@pk-n Your approach solves the problem related to table provision exceeding but doesn't solve issues listed above.

I would consider raising an exception with unprocessed items

[pk-n]

Thank you @andrykonchin for replying to my question.
It makes sense. Now I am handling all the six points out side(in my code) the gem where I make a call to batch operation.

[pboling]

Accidentally closed. Should we leave this open for further discussion of ways we can mitigate some of these inside the gem? I like the idea of raising a custom error that has the unprocessed items embedded.

[andrykonchin]

@pboling
As far as I understand the issue is - the gem can handle unprocessed items (due to table provision limits exceeding) itself and make several attempts to put unprocessed items and use recommended exponential backoff algorithm.

It's reasonable behavior. Not default of course. I will work on it.

For another thing batch_write_item doesn't return unprocessed items now and a client can't handle them himself. So by default it should at least return something or raise an exception with attached unprocessed items

[pboling]

In order to internalize an exponential backoff solution within the gem we would need to have something like celluloid to fork those attempts and return the original control to the caller, otherwise calls to Dynamoid could take exponentially long. Exponential backoff isn't a good idea for inline processing.

[andrykonchin]

@pboling Concurrency is a good approach but it makes things more difficult and looks like users need at least some basic solution to handle table provision limits exceeding.

E.g. we may implement following:

• default behavior is to raise exception with unhandled items or just return them
• user may configure exponential/constant backoff globally/locally - attempts count/timeout and delay
• after exceeding specified attempts count/timeout gem will either raise exception or return remaining unprocessed items

@pk-n what do you think?

[andrykonchin]

@pboling @pk-n

Ping

[pboling]

I think having an internal backoff feature would be a good idea for a plugin gem, like dynamoid-backoff. That way people who don't need the extra feature, because they have already implemented the exponential backoff as suggested by Amazon, won't have it complicating their codebase. It would also be a good way to force us to think about the public surface of the gem for writing plugins.