Mongo::Error::OperationFailure: exception: Btree::insert: key too large to index, failing errbit.problems.$message_1 1267

[hoffi]

Looks like the same issue as #1001.

We got the error:

Mongo::Error::OperationFailure: exception: Btree::insert: key too large to index, failing errbit.problems.$message_1 1267

for an exception with a very long (> 1000 chars) message. Stacktrace is:

app/models/problem.rb:82 → cache_notice
app/models/error_report.rb:80 → cache_attributes_on_problem
app/models/error_report.rb:58 → generate_notice!
app/controllers/notices_controller.rb:15 → create

We are on commit 422d583 with Ruby 2.2.5

[stevecrozz]

Thanks for the report. So far I haven't been able to reproduce this in a unit test because saving a notice truncates its message and that happens before the cache_notice method is called. That's the way its supposed to work. I'll try calling it in a real environment soon with a big notice. If you're able to write a failing unit test that could help a lot, but I'm sure I can figure it out eventually either way :)

[hoffi]

Hi i also tried to reproduce it with a clean install. It only seem to fail with a special message.
In my case:

Elasticsearch::Transport::Transport::Errors::InternalServerError: [500] {"error":"SearchPhaseExecutionException[Failed to execute phase [query_fetch], all shards failed; shardFailures {[abc][test][0]: QueryPhaseExecutionException[[test][0]: query[function score (_all:t4t44äöäöäööäöäöäöäöäälüöläpläfdälfäpdlsfaäpldspsadpfäsdkfasdöäkfadsökfjaädsfjsdaäfjadsklfldslsäfjkläsdajfläaslhfldskhfasljdhfl44444t44t4t4t4t44t4444tt444tt4þt444t4gt4t444t44t444g4444t4g44g4tt444g4444tgt444ggþ444þ4t4þ4t44444t4444g4444t44gþt4t4tþg4t44t4t4444gt44t444t4t4t444tt44t44þt4t4þt4444444þgþ4tt4t4g444gt4t4t444þ44g4t44g4tgþ4t4t444t4þþ444t44t4t44~2,function=script[_score * _source.boost], params [null])],from[0],size[10]: Query Failed [Failed to execute main query]]; nested: RuntimeException[org.apache.lucene.util.automaton.TooComplexToDeterminizeException: Determinizing automaton would result in more than 10000 states.]; nested: TooComplexToDeterminizeException[Determinizing automaton would result in more than 10000 states.]; }]","status":500}

However this message (also over 1000 chars) does work:

Elasticsearch::Transport::Transport::Errors::InternalServerError: [500] {"error":"SearchPhaseExecutionException[Failed to execute phase [query_fetch], all shards failed; shardFailures {[abc][test][0]: QueryPhaseExecutionException[[test][0]: query[function score (_all:TEST123TESTÖÄÜþLorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, conset~2,function=script[_score * _source.boost], params [null])],from[0],size[10]: Query Failed [Failed to execute main query]]; nested: RuntimeException[org.apache.lucene.util.automaton.TooComplexToDeterminizeException: Determinizing automaton would result in more than 10000 states.]; nested: TooComplexToDeterminizeException[Determinizing automaton would result in more than10000 states.]; }]","status":500}

And this one fails:

Elasticsearch::Transport::Transport::Errors::InternalServerError: [500] {"error":"SearchPhaseExecutionException[Failed to execute phase [query_fetch], all shards failed; shardFailures {[abc][test][0]: QueryPhaseExecutionException[[test][0]: query[function score (_all:TEST123TESTÖÄÜLorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam eraþþþþþþþþþþþþþþþþþþþþþþþþþþþþþþt, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor þþþsit amet. Lorem ipsum dolor sit amet, conset44~2,function=script[_score * _source.boost], params [null])],from[0],size[10]: Query Failed [Failed to execute main query]]; nested: RuntimeException[org.apache.lucene.util.automaton.TooComplexToDeterminizeException: Determinizing automaton would result in more than 10000 states.]; nested: TooComplexToDeterminizeException[Determinizing automaton would

Seems like it has something to do with this "þ" in the error messages.

[hoffi]

I can reproduce it in a unit test if you change the long_message variable inside the unit test in spec/models/notice_spec.rb to one of the problematic messages from my last comment.

Example:

let(:long_es_message) do
  'Elasticsearch::Transport::Transport::Errors::InternalServerError: ' \
  '[500] {"error":"SearchPhaseExecutionException[Failed to execute phase ' \
  '[query_fetch], all shards failed; shardFailures {[abc][test][0]: ' \
  'QueryPhaseExecutionException[[test][0]: query[function score ' \
  '(_all:t4t44äöäöäööäöäöäöäöäälüöläpläfdälfäpdlsfaäpldspsadpfäsdkfasdö' \
  'äkfadsökfjaädsfjsdaäfjadsklfldslsäfjkläsdajfläaslhfldskhfasljdhfl444' \
  '44t44t4t4t4t44t4444tt444tt4þt444t4gt4t444t44t444g4444t4g44g4tt444g44' \
  '44tgt444ggþ444þ4t4þ4t44444t4444g4444t44gþt4t4tþg4t44t4t4444gt44t444t' \
  '4t4t444tt44t44þt4t4þt4444444þgþ4tt4t4g444gt4t4t444þ44g4t44g4tgþ4t4t4' \
  '44t4þþ444t44t4t44~2,function=script[_score * _source.boost], params ' \
  '[null])],from[0],size[10]: Query Failed [Failed to execute main ' \
  'query]]; nested: RuntimeException[org.apache.lucene.util.automaton.' \
  'TooComplexToDeterminizeException: Determinizing automaton would ' \
  'result in more than 10000 states.]; nested: TooComplexToDeterminize' \
  'Exception[Determinizing automaton would result in more than 10000 ' \
  'states.]; }]","status":500}'
end

it 'truncates the long es message' do
  notice = Fabricate(:notice, message: long_es_message)
  expect(long_es_message.length).to be > 1000
  expect(notice.message.length).to eq 1000
end

$ bundle exec rspec spec/models/notice_spec.rb
Randomized with seed 7446
....F................

Failures:

  1) Notice#message= truncates the long es message
     Failure/Error: notice = Fabricate(:notice, message: long_es_message)
     Mongo::Error::OperationFailure:
       exception: Btree::insert: key too large to index, failing errbit_test.problems.$message_1 1060 { : "Elasticsearch::Transport::Transport::Errors::InternalServerError: [500] {"error":"SearchPhaseExecutionException[Failed to execute phase [query_fetch],..." } (17280)
     # ./app/models/problem.rb:82:in `cache_notice'
     # ./spec/fabricators/notice_fabricator.rb:12:in `block (2 levels) in <top (required)>'
     # ./spec/models/notice_spec.rb:52:in `block (3 levels) in <top (required)>'

Finished in 0.69611 seconds (files took 3.62 seconds to load)
21 examples, 1 failure

Failed examples:

rspec ./spec/models/notice_spec.rb:51 # Notice#message= truncates the long es message