New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] when is exceeded the index key limit in mongo #1289

Closed
iariasleon opened this Issue Sep 25, 2015 · 25 comments

Comments

Projects
None yet
5 participants
@iariasleon
Copy link
Contributor

iariasleon commented Sep 25, 2015

the index key limit in mongo is 1024, see issue #1283

if type or id exceed index key limit ,CB responds 201 and nothing is stored in mongo.

Request:

 POST http://qa-orion-fe-01:1026/v2/entities
  Content-Type: application/json
  Fiware-Service: test_entities_type
  Fiware-ServicePath: /test
  {"type": "TZQIpnz4YX7NgOo2DCSJ6llpviu6mG2Xc2H7MqPaJy9fGRtUcjg1UmguN39pwSKkFJpIJJ1n4RGwjJBkk5iGDl7OY1RW5dZRKQ6hKCSSufXZqm02ECdoqQWnjR6WZ5PGJMRpSmnrVbdZMKfXkJ04WiCUekxiLxgHM6ToydSt8XZ2yDdYcXpXBSVOx66bIi8CoImJAqLYc39apAF77hI1Y1RPksM1S8z9d5sVp0IxhnFN27Al72XXQDqwSTW7YUZyJJeXzM9OI4lQY0CnZLignlNqxiW8Hr4SfZrqWhailwIZ9Q6FyWfjhcSrWK2RF89kVkGIt9DuO7yaGQ5ONCessB2f8rab0CVvVC6WGafUtvDJs8raCQfEzMnLZ2PW6jDMFH1RXPAoxRN78Ho1QLgJB4YVqUWjgB4ZMQCWFQbaG0fSMAImLyzwqjSAoIbobooQzKjaL6aZzIYV1HlVsZSzrnlz68cHD62ZQxIjxsTjPjk2jlXAPCWi23KO4XOB3x9NrzYN2Eh0aN4VDCHfnFQWj4LkqoF2LbZLYonLLrty9yLRqUD1Sq4dpHO52HTWI6dh9LZI0oT4spnPHNmlnsifjatn3e6hmH6gFiXiyXKdndptaHUVaI8GwDr6zeQH0GYlfw4Ho46iaECauPbpgD9YONA0kAl8BFZpL83yz3pJyhRNK4vCWHS676PBEGr8GXBkwTScge5f2pACnvfNmHbrPtm1FHuZ6ZRH318hBt7p98XSF757lz3ML2ixoecLBORRTBgHto95feclNIyrbRgZ9LiOfQrifoagfhGh04eLN6dPCt3mXDmirYR6kU2xUGOmGug4QM1OKUyWUK8vPCqfG4iu4Klt9Pv2s1ttPHmloeEvZ580b7AODyxQqWhHbVJ0OmAJ02h14ONdkvGKDBBzcmh5cFiTDMZDRMFU88NnTQJkFoEBjtz6OYkRdR2U1ctpPXyYkFP0xednTNl6cC9oRz7vXiOXFDozYFPV3lZqeenIsPlQhFpqim11DxTdEbqM", "temperature": "34", "id": "room_1"}

Response

http code: 201
 content-length: 0

Log

time=2015-09-25T08:56:09.696CEST | lvl=INFO | trans=1443164167-603-00000000003 | function=processContextElement | comp=Orion | msg=MongoCommonUpdate.cpp[2328]: Database Operation Successful ({ _id.id: "room_1", _id.type: "TZQIpnz4YX7NgOo2DCSJ6llpviu6mG2Xc2H7MqPaJy9fGRtUcjg1UmguN39pwSKkFJpIJJ1n4RGwjJBkk5iGDl7OY1RW5dZRKQ6hKCSSufXZqm02ECdoqQWnjR6WZ5PGJMRpSmnrVbdZMKfXkJ04Wi...", _id.servicePath: /^\/test$/ })
time=2015-09-25T08:56:09.696CEST | lvl=INFO | trans=1443164167-603-00000000003 | function=processContextElement | comp=Orion | msg=MongoCommonUpdate.cpp[2388]: Database Operation Successful ({ _id.id: "room_1", _id.type: "TZQIpnz4YX7NgOo2DCSJ6llpviu6mG2Xc2H7MqPaJy9fGRtUcjg1UmguN39pwSKkFJpIJJ1n4RGwjJBkk5iGDl7OY1RW5dZRKQ6hKCSSufXZqm02ECdoqQWnjR6WZ5PGJMRpSmnrVbdZMKfXkJ04Wi...", _id.servicePath: /^\/test$/ })
time=2015-09-25T08:56:09.698CEST | lvl=ERROR | trans=1443164167-603-00000000003 | function=createEntity | comp=Orion | msg=MongoCommonUpdate.cpp[1993]: Database Error (Database Error: collection: orion.entities - insert(): { _id: { id: "room_1", type: "TZQIpnz4YX7NgOo2DCSJ6llpviu6mG2Xc2H7MqPaJy9fGRtUcjg1UmguN39pwSKkFJpIJJ1n4RGwjJBkk5iGDl7OY1RW5dZRKQ6hKCSSufXZqm02ECdoqQWnjR6WZ5PGJMRpSmnrVbdZMKfXkJ04Wi...", servicePath: "/test" }, attrNames: [ "temperature" ], attrs: { temperature: { type: "", creDate: 1443164169, modDate: 1443164169, value: "34" } }, creDate: 1443164169, modDate: 1443164169 } - exception: OperationException: { index: 0, code: 17280, errmsg: "insertDocument :: caused by :: 17280 Btree::insert: key too large to index, failing orion.entities.$_id_ 1086 { : { id: "room_1", type: "TZQIpnz4YX7Ng...", op: { _id: { id: "room_1", type: "TZQIpnz4YX7NgOo2DCSJ6llpviu6mG2Xc2H7MqPaJy9fGRtUcjg1UmguN39pwSKkFJpIJJ1n4RGwjJBkk5iGDl7OY1RW5dZRKQ6hKCSSufXZqm02ECdoqQWnjR6WZ5PGJMRpSmnrVbdZMKfXkJ04Wi...", servicePath: "/test" }, attrNames: [ "temperature" ], attrs: { temperature: { type: "", creDate: 1443164169, modDate: 1443164169, value: "34" } }, creDate: 1443164169, modDate: 1443164169 } })

@iariasleon iariasleon added this to the 0.25.0 milestone Sep 25, 2015

@kzangeli

This comment has been minimized.

Copy link
Member

kzangeli commented Sep 25, 2015

Interesting ...
We'll have to impose some restriction and return an error.
strlen(id) + strlen(type) + some-constant > 1024 => ERROR?

@iariasleon

This comment has been minimized.

Copy link
Contributor Author

iariasleon commented Sep 25, 2015

service path should also be included

@fortizc fortizc self-assigned this Oct 7, 2015

@fortizc

This comment has been minimized.

Copy link
Contributor

fortizc commented Oct 8, 2015

What if the restriction for 1024 is implemented in the mongoUpdateContext.cpp ?

@fgalan

This comment has been minimized.

Copy link
Member

fgalan commented Oct 8, 2015

What if the restriction for 1024 is implemented in the mongoUpdateContext.cpp ?

Pros:

  • Given that the problem is related with a DB limitation, probably is the layer where that check belongs.

Cons:

  • mongoBackend operations run protected within a take/give sempahore area (except if -reqMutexPolicy is used), i.e. only one thread runs inside the mongoBackend. Thus, it is better to put checks that don't depend on DB results (as the ones that depend only of the request input, as this one) outside mongoBackend.
@fortizc

This comment has been minimized.

Copy link
Contributor

fortizc commented Oct 8, 2015

Ok, now I check mongoCommonUpdate.cpp in line 2027, the error is captured here in createEntity function. I can use it?

See https://github.com/telefonicaid/fiware-orion/blob/develop/src/lib/mongoBackend/MongoCommonUpdate.cpp#L2027

@fgalan

This comment has been minimized.

Copy link
Member

fgalan commented Oct 16, 2015

Ok, now I check mongoCommonUpdate.cpp in line 2027, the error is captured here in createEntity function. I can use it?

It seem that the code has "moved" in develop branch (is no longer pointing inside createEntity method)... so I'm not sure of the reference.

Hint: to get "inmutable" references, use a commits in the develop branch instead of the develop branch itself (which point to HEAD commit), e.g.

/* First CPr lookup (in the case some CER is not found): looking in E-A registrations */

@fortizc

This comment has been minimized.

Copy link
Contributor

fortizc commented Oct 19, 2015

I pull the latest commit, compile and test with the case posted by iariasleon and surprise! it works... can anybody confirm this?

@iariasleon

This comment has been minimized.

Copy link
Contributor Author

iariasleon commented Oct 27, 2015

@fortizc
What http code is returned?
What are the error and the description messages expected ?

@fortizc

This comment has been minimized.

Copy link
Contributor

fortizc commented Oct 27, 2015

@iariasleon
Sorry but the last days I'm not working in this bug, but I will return from tomorrow

@fortizc

This comment has been minimized.

Copy link
Contributor

fortizc commented Oct 27, 2015

@kzangeli what should be the constant value?

@fgalan

This comment has been minimized.

Copy link
Member

fgalan commented Oct 27, 2015

Actually, the formula should be:

strlen(id) + strlen(type) + strlen(servicePath) + some-constant > 1024 => ERROR

Not sure about "some-constant". I guess that it depends on how much additional space it used in the embedded document identified by _id (i.e. {id: ..., type: ..., servicePath: ...}) appart from the 3 subfields themselves.

@iariasleon , have your test bring some light about which value "some-constant" could take?

@fgalan

This comment has been minimized.

@fgalan

This comment has been minimized.

Copy link
Member

fgalan commented Oct 28, 2015

While we get the right answer, in order not be blocked, I'd suggest to assume a reasonable large value for "some-constant". Maybe 100 is a good heuristic (100 is around 10% of max lenght 1024).

@iariasleon

This comment has been minimized.

Copy link
Contributor Author

iariasleon commented Oct 28, 2015

I think, first is to define the behaviour, see #1283
and second the max length is 1024 bytes, no chars.

@fgalan

This comment has been minimized.

Copy link
Member

fgalan commented Oct 28, 2015

Just to clarify things:

#1283 is related with the lengths of the indivitual fields id, type and servicePath. Unfortunatelly, we don't have such information (the decission is a functional requirement and the development team -i.e. us- don't have the ownership to define such requirement).

Good news are that we don't need solving #1283 to solve this (#1289) issue. Otherwise, we would be blocked! This (#1289) only needs to define a limit in the sum of the field. The limit should be:

strlen(id) + strlen(type) + strlen(servicePath) + C < 1024

We don't know the value of C (question at http://stackoverflow.com/questions/33377441/mongodb-check-length-when-using-embedded-documents-for-id-index is about trying to find it). In abcense of an answer, we have to do some guessing. I think that C = 100 is a good guessing, but if you have another suggestion, please go ahead.

If after reading this explantion is not 100% clear how this (#1289) has to be solved, please contact me at skype.

@fgalan

This comment has been minimized.

Copy link
Member

fgalan commented Oct 28, 2015

According to #1286 "rules" for NGSIv2 errors, I'd say that this case should be:

400 - Bad Request {"error": "BadRequest", "description": "Too long entity id/type/servicePath combination" }

Don't forget to include a .test to cover the case! ;)

fortizc added a commit to fortizc/fiware-orion that referenced this issue Oct 28, 2015

fortizc added a commit to fortizc/fiware-orion that referenced this issue Oct 28, 2015

@fortizc

This comment has been minimized.

Copy link
Contributor

fortizc commented Oct 28, 2015

Pull request sent

@fortizc fortizc referenced this issue Oct 28, 2015

Merged

Bug #1289 #1441

fortizc added a commit to fortizc/fiware-orion that referenced this issue Oct 28, 2015

fortizc added a commit to fortizc/fiware-orion that referenced this issue Oct 28, 2015

fortizc added a commit to fortizc/fiware-orion that referenced this issue Oct 28, 2015

fortizc added a commit to fortizc/fiware-orion that referenced this issue Oct 29, 2015

fortizc added a commit to fortizc/fiware-orion that referenced this issue Oct 29, 2015

fortizc added a commit to fortizc/fiware-orion that referenced this issue Oct 29, 2015

fortizc added a commit to fortizc/fiware-orion that referenced this issue Oct 29, 2015

fortizc added a commit to fortizc/fiware-orion that referenced this issue Oct 29, 2015

fortizc added a commit to fortizc/fiware-orion that referenced this issue Oct 29, 2015

fortizc added a commit to fortizc/fiware-orion that referenced this issue Oct 29, 2015

fortizc added a commit to fortizc/fiware-orion that referenced this issue Oct 29, 2015

fortizc added a commit to fortizc/fiware-orion that referenced this issue Oct 29, 2015

@fgalan fgalan closed this in #1441 Oct 29, 2015

fgalan pushed a commit that referenced this issue Oct 29, 2015

@fgalan fgalan reopened this Oct 29, 2015

@fgalan

This comment has been minimized.

Copy link
Member

fgalan commented Oct 29, 2015

Fixed in PR #1441. @iariasleon please have a look and close it if you find correct.

@iariasleon

This comment has been minimized.

Copy link
Contributor Author

iariasleon commented Nov 2, 2015

LGTM

http code: 400
{
     "error":"BadRequest",
     "description":"Too long entity id/type/servicePath combination"
}

@iariasleon iariasleon closed this Nov 2, 2015

@mauze-muawwaz-elastica

This comment has been minimized.

Copy link

mauze-muawwaz-elastica commented Mar 9, 2016

How do I figure out programatically (if possible) that the insertion I'm going to perform will throw this error?
In my case, there is an indexed field on a collection which doesn't allow me to save a value greater than 1007. The documentation says 1024, where are those extra 17 bytes used?
Is there any formula which can help me identify this prior to insertion?

@fgalan fgalan modified the milestones: 1.0.0, 0.25.0 Mar 9, 2016

@fgalan fgalan assigned kzangeli and unassigned iariasleon Mar 9, 2016

@fgalan

This comment has been minimized.

Copy link
Member

fgalan commented Mar 9, 2016

The "formula" being used should be in the code. @kzangeli , @fortizc , could you help to locate it an add it as comment to this issue, please?

(Reopening issue while the answer is in progress)

@fgalan fgalan reopened this Mar 9, 2016

@kzangeli

This comment has been minimized.

Copy link
Member

kzangeli commented Mar 9, 2016

In postEntities.cpp, the function legalEntityLength determines if the total is too long:

static bool legalEntityLength(Entity* eP, const std::string& servicePath)
{
  return (servicePath.size() +
          eP->id.size()      +
          eP->type.size()    +
          STRUCTURAL_OVERHEAD_BSON_ID) < 1024;
}

// servicePath.size:    the string-length of the service path
// eP->id.size() :      the string-length of the entity id
// eP->type.size();     the string-length of the entity type
//
// STRUCTURAL_OVERHEAD_BSON_ID is 10
//
// Now, if the sum of these 4 sizes is less than 1024, then all is OK
@mauze-muawwaz-elastica

This comment has been minimized.

Copy link

mauze-muawwaz-elastica commented Mar 9, 2016

Thanks.

@fgalan

This comment has been minimized.

Copy link
Member

fgalan commented Mar 9, 2016

Maybe we should put the rule in the documentation (although I'm not sure in which section...)

@fgalan

This comment has been minimized.

Copy link
Member

fgalan commented Mar 11, 2016

At the end "Known limitations" seems to be a good candidate section in the documentation. Fixed in PR #1899

@fgalan fgalan closed this Mar 11, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment