API occasionally sends a malformed query and Mongo closes connection

[fra967]

This issue appears occasionally in an API production environment used by Publish

MongoDB log reports an error and the database closes the connection, causing a sequence of API errors and data loss

The error reported by MongoDB is:

Assertion: 10307:Client Error: bad object in message: invalid bson type in object with unknown _id
AssertionException handling request, closing client connection: 10307 Client Error: bad object in message: invalid bson type in object with unknown _id

Being a live environment, it is difficult to trace which query creates this issue, but it has been seen after PUT or DELETE requests.

Probably the first step would be to introduce verbose logging (with the full content of the request sent to the database) in order to gather more info

API 1.15.5
Node 6.10.2

[jimlambie]

@fra967 what is th mongodb version?

[fra967]

MongoDB 3.2.10 (it's a replicaset)

[fra967]

I managed to trace the offending query, which is probably sent by a badly written hook, yet to be identified

It's a simple empty _id: { _id: }

If I try to hit API with a similar query, it does not generate the MongoDB error

Maybe there is some validation, which doesn't happen when the query is sent directly from the model?

[jimlambie]

The error originates from a hook that passes API a query containing an ObjectID. API attempts to make this case insensitive and it looks as if a case insensitive ObjectID = "".... The hook can be changed to pass a string in the query instead, as a temporary fix.

Two pieces of work:

1. make the case-insensitive method ignore internal fields such as _id
2. include a check inside the relevant model operation that determines if the call came from a hook and if it does, skip the hook step.

[jimlambie]

Closed by d103f62