New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

apiVersion is and-ed to all queries and impacts performance #211

Closed
fra967 opened this Issue Mar 8, 2017 · 5 comments

Comments

Projects
None yet
2 participants
@fra967
Copy link
Member

fra967 commented Mar 8, 2017

The configuration paramenter useVersionFilter can control whether apiVersion is passed in all db queries (by default is set to true).

However it seems that it is necessary for composition to work, and therefore has a negative impact on performance.

In the meantime, it could be useful that API enforces indexes on apiVersion on all collections

@fra967

This comment has been minimized.

Copy link
Member Author

fra967 commented Apr 27, 2017

I have to request to prioritise this issue. Despite being indexed, the impact of and-ing a string to all queries is significant for Publish operations, with an estimate of 3-4 additional seconds for each operation, in particular in production environments where some collections contain up to 700k documents.

@fra967

This comment has been minimized.

Copy link
Member Author

fra967 commented Apr 28, 2017

Another example of the impact of apiVersion

Count queries that would normally be almost immediate (few milliseconds) end up being executed as db.collection.find({apiVersion:"1.0"}).count() which triggers a COUNT_SCAN and can take up to few seconds, despite apiVersion being indexed,

@jimlambie

This comment has been minimized.

Copy link
Member

jimlambie commented Apr 28, 2017

@fra967

This comment has been minimized.

Copy link
Member Author

fra967 commented May 10, 2017

We are having some errors with composition with apiVersionFilter = false
Unfortunately it seems very random and at present un-reproduceable, however it happens only with this change and with the upgrade from 1.15.5 to 1.16.3

@fra967 fra967 reopened this May 10, 2017

@fra967

This comment has been minimized.

Copy link
Member Author

fra967 commented May 11, 2017

More details about the above errors
Running Publish (latest from master) with a client dataset

The article 587cdcbc80222c9e7267ace7 is syndicated under 20-something publications

Using API 1.15.5, the article syndication information is shown in Publish

Using API 1.16.3, the category does not appear

The sequence of calls with API 1.15.5 is

GET/1.0/radio/articles?compose=true&count=20&filter=%7B%22_id%22%3A%22587cdcbc80222c9e7267ace7%22%7D&page=1&sort=%7B%22createdAt%22%3A-1%7D
GET/1.0/radio/authors?compose=true&count=1000&filter=%7B%22publish_id%22%3A%229657e7c3-d542-48de-a00f-0af9e1321ec8%22%7D&page=1
GET/1.0/variables
GET/1.0/variables
GET/1.0/variables
GET/1.0/radio/tracking_pixels?compose=true&count=0&fields=%7B%22name%22%3A1%2C%22trackingPixel%22%3A1%7D&filter=%7B%7D&page=1
GET/1.0/radio/sponsors?compose=true&count=0&fields=%7B%22name%22%3A1%2C%22images%22%3A1%7D&filter=%7B%7D&page=1
GET/1.0/radio/authors?compose=true&count=0&fields=%7B%22fullname%22%3A1%7D&filter=%7B%7D&page=1
GET/1.0/radio/tags?compose=true&count=40&fields=%7B%22tag%22%3A1%7D&filter=%7B%22tag.name%22%3A%7B%22%24regex%22%3A%22%22%7D%7D&page=1
GET/1.0/radio/presenters?compose=true&count=0&fields=%7B%22name%22%3A1%2C%22url%22%3A1%2C%22furl%22%3A1%7D&filter=%7B%7D&page=1
GET/1.0/radio/publications?compose=true&count=200&fields=%7B%22_id%22%3A1%2C%22name%22%3A1%2C%22displayField%22%3A1%2C%22categories%22%3A1%7D&filter=%7B%7D&page=1&sort=%7B%22name%22%3A1%7D
GET/1.0/variables
GET/1.0/radio/syndicate_groups?compose=true&count=100&fields=%7B%22_id%22%3A1%2C%22name%22%3A1%2C%22displayField%22%3A1%2C%22publications%22%3A1%7D&filter=%7B%7D&page=1&sort=%7B%22name%22%3A1%7D
GET/1.0/variables
GET/1.0/radio/publications?compose=true&count=1&filter=%7B%22_id%22%3A%22587cb67780222c9e7266ca90%22%7D&page=1&sort=%7B%22name%22%3A1%7D
GET/1.0/variables
GET/1.0/radio/publications?compose=true&count=1&filter=%7B%22_id%22%3A%22587cb67780222c9e7266ca90%22%7D&page=1&sort=%7B%22name%22%3A1%7D
GET/1.0/radio/categories?compose=true&count=1&filter=%7B%22_id%22%3A%22587cb5cb80222c9e7266bb37%22%7D&page=1&sort=%7B%22name%22%3A1%7D
GET/1.0/variables
GET/1.0/radio/categories?compose=true&count=1&filter=%7B%22_id%22%3A%22587cb5cb80222c9e7266bb37%22%7D&page=1&sort=%7B%22name%22%3A1%7D
GET/1.0/variables
GET/1.0/variables
GET/1.0/variables

the same page, with the same version of Publish and the same dataset, but with API 1.16.3

GET/1.0/radio/articles?compose=true&count=20&filter=%7B%22_id%22%3A%22587cdcbc80222c9e7267ace7%22%7D&page=1&sort=%7B%22createdAt%22%3A-1%7D
GET/1.0/radio/authors?compose=true&count=1000&filter=%7B%22publish_id%22%3A%229657e7c3-d542-48de-a00f-0af9e1321ec8%22%7D&page=1
GET/1.0/variables
GET/1.0/variables
GET/1.0/variables
GET/1.0/radio/tracking_pixels?compose=true&count=0&fields=%7B%22name%22%3A1%2C%22trackingPixel%22%3A1%7D&filter=%7B%7D&page=1
GET/1.0/radio/tags?compose=true&count=40&fields=%7B%22tag%22%3A1%7D&filter=%7B%22tag.name%22%3A%7B%22%24regex%22%3A%22%22%7D%7D&page=1
GET/1.0/radio/sponsors?compose=true&count=0&fields=%7B%22name%22%3A1%2C%22images%22%3A1%7D&filter=%7B%7D&page=1
GET/1.0/radio/presenters?compose=true&count=0&fields=%7B%22name%22%3A1%2C%22url%22%3A1%2C%22furl%22%3A1%7D&filter=%7B%7D&page=1
GET/1.0/radio/authors?compose=true&count=0&fields=%7B%22fullname%22%3A1%7D&filter=%7B%7D&page=1
GET/1.0/radio/publications?compose=true&count=200&fields=%7B%22_id%22%3A1%2C%22name%22%3A1%2C%22displayField%22%3A1%2C%22categories%22%3A1%7D&filter=%7B%7D&page=1&sort=%7B%22name%22%3A1%7D
GET/1.0/variables
GET/1.0/radio/syndicate_groups?compose=true&count=100&fields=%7B%22_id%22%3A1%2C%22name%22%3A1%2C%22displayField%22%3A1%2C%22publications%22%3A1%7D&filter=%7B%7D&page=1&sort=%7B%22name%22%3A1%7D
GET/1.0/variables
GET/1.0/variables
GET/1.0/variables
GET/1.0/variables
GET/1.0/variables
GET/1.0/variables

Note that after the call to syndicate_groups, nothing else happens, and neither the publication nor the category is looked up

However up to that moment the calls to API return exactly the same data in 1.15.5 and 1.16.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment