Limit should not be incremented when paging

[rosswilson]

The limit argument should not be incremented while paging. I'm using Ecto queries to grab the data, but I think the limit param should not increment regardless of what backing-store someone is using.

With bulk_page_size set to 1000, the following SQL queries were being made.

SELECT a0."id", a0."barcode" FROM "assets" AS a0 ORDER BY a0."id" LIMIT $1 OFFSET $2 [1000, 0]
SELECT a0."id", a0."barcode" FROM "assets" AS a0 ORDER BY a0."id" LIMIT $1 OFFSET $2 [2000, 1000]
SELECT a0."id", a0."barcode" FROM "assets" AS a0 ORDER BY a0."id" LIMIT $1 OFFSET $2 [3000, 2000]

Which results in overlapping pages:

• First query returns items 0 to 1000
• Second query returns items 1000 to 3000
• Third query returns items 2000 to 5000

Items 2000 to 3000 were being indexed twice, with ElasticSearch throwing a version conflict, document already exists (current version [1]) error.

[coveralls]

Coverage decreased (-3.7%) to 68.783% when pulling dff6350 on rosswilson:master into 1470043 on infinitered:master.

[rosswilson]

I get a 404 when accessing the Semaphore CI build page, so I can't check the tests. Is it locked down somehow?

[danielberkompas]

Yeah, I should really move this project to Travis so that the build failure will be public. Are the tests passing for you locally? This what we're getting on Semaphore:

 1) test doc at Elasticsearch.Index.hot_swap/4 (7) (Elasticsearch.IndexTest)
     test/elasticsearch/indexing/index_test.exs:6
     ** (ExUnit.TimeoutError) test timed out after 60000ms. You can change the timeout:
     
       1. per test by setting "@tag timeout: x"
       2. per case by setting "@moduletag timeout: x"
       3. globally via "ExUnit.start(timeout: x)" configuration
       4. or set it to infinity per run by calling "mix test --trace"
          (useful when using IEx.pry)
     
     Timeouts are given as integers in milliseconds.
     
     stacktrace:
       lib/poison/encoder.ex:85: Poison.Encoder.BitString.encode/2
       lib/poison/encoder.ex:227: anonymous fn/4 in Poison.Encoder.Map.encode/3
       lib/poison/encoder.ex:228: Poison.Encoder.Map."-encode/3-lists^foldl/2-0-"/3
       lib/poison/encoder.ex:228: Poison.Encoder.Map.encode/3
       lib/poison/encoder.ex:227: anonymous fn/4 in Poison.Encoder.Map.encode/3
       lib/poison/encoder.ex:228: Poison.Encoder.Map."-encode/3-lists^foldl/2-0-"/3
       lib/poison/encoder.ex:228: Poison.Encoder.Map.encode/3
       lib/poison.ex:41: Poison.encode!/2

............

  2) test .run only keeps two index versions (Mix.Tasks.Elasticsearch.BuildTest)
     test/mix/elasticsearch.build_test.exs:45
     ** (ExUnit.TimeoutError) test timed out after 60000ms. You can change the timeout:
     
       1. per test by setting "@tag timeout: x"
       2. per case by setting "@moduletag timeout: x"
       3. globally via "ExUnit.start(timeout: x)" configuration
       4. or set it to infinity per run by calling "mix test --trace"
          (useful when using IEx.pry)
     
     Timeouts are given as integers in milliseconds.
     
     stacktrace:
       lib/poison/parser.ex:274: Poison.Parser.skip_whitespace/2
       lib/poison/parser.ex:65: Poison.Parser.value/3
       lib/poison/parser.ex:90: Poison.Parser.object_pairs/4
       lib/poison/parser.ex:124: Poison.Parser.array_values/4
       lib/poison/parser.ex:90: Poison.Parser.object_pairs/4
       lib/poison/parser.ex:37: Poison.Parser.parse/2
       lib/poison/parser.ex:51: Poison.Parser.parse!/2
       lib/poison.ex:83: Poison.decode!/2



  3) test .run builds configured index (Mix.Tasks.Elasticsearch.BuildTest)
     test/mix/elasticsearch.build_test.exs:38
     ** (ExUnit.TimeoutError) test timed out after 60000ms. You can change the timeout:
     
       1. per test by setting "@tag timeout: x"
       2. per case by setting "@moduletag timeout: x"
       3. globally via "ExUnit.start(timeout: x)" configuration
       4. or set it to infinity per run by calling "mix test --trace"
          (useful when using IEx.pry)
     
     Timeouts are given as integers in milliseconds.
     
     stacktrace:
       lib/poison/encoder.ex:66: Poison.Encoder.impl_for/1
       lib/poison/encoder.ex:66: Poison.Encoder.impl_for!/1
       lib/poison/encoder.ex:69: Poison.Encoder.encode/2
       lib/poison/encoder.ex:227: anonymous fn/4 in Poison.Encoder.Map.encode/3
       lib/poison/encoder.ex:228: Poison.Encoder.Map."-encode/3-lists^foldl/2-0-"/3
       lib/poison/encoder.ex:228: Poison.Encoder.Map.encode/3
       lib/poison.ex:41: Poison.encode!/2
       (elasticsearch) lib/elasticsearch/indexing/bulk.ex:60: Elasticsearch.Index.Bulk.encode!/2



  4) test .run --existing checks if index exists (Mix.Tasks.Elasticsearch.BuildTest)
     test/mix/elasticsearch.build_test.exs:59
     ** (ExUnit.TimeoutError) test timed out after 60000ms. You can change the timeout:
     
       1. per test by setting "@tag timeout: x"
       2. per case by setting "@moduletag timeout: x"
       3. globally via "ExUnit.start(timeout: x)" configuration
       4. or set it to infinity per run by calling "mix test --trace"
          (useful when using IEx.pry)
     
     Timeouts are given as integers in milliseconds.
     
     stacktrace:
       :prim_inet.recv0/3
       (hackney) /home/runner/elasticsearch-elixir/deps/hackney/src/hackney_response.erl:76: :hackney_response.wait_status/1
       (hackney) /home/runner/elasticsearch-elixir/deps/hackney/src/hackney.erl:358: :hackney.send_request/2
       (httpoison) lib/httpoison/base.ex:439: HTTPoison.Base.request/9
       (elasticsearch) lib/elasticsearch.ex:196: Elasticsearch.put/3
       (elixir) lib/stream.ex:516: anonymous fn/4 in Stream.map/2
       (elixir) lib/stream.ex:1413: anonymous fn/3 in Enumerable.Stream.reduce/3
       (elixir) lib/stream.ex:228: anonymous fn/4 in Stream.chunk_while/4

......

Finished in 242.0 seconds
44 tests, 4 failures

[danielberkompas]

Sorry for the delay in getting this fixed. I've released a new version: 0.1.1