Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[V3 Config] Update Mongo document organization to bypass doc size restriction #2536

Merged
merged 16 commits into from Apr 3, 2019

Conversation

@tekulvw
Copy link
Member

commented Mar 31, 2019

Type

  • Bugfix
  • Enhancement
  • New feature

Description of the changes

This PR reorganizes how the Mongo driver stores data on the database. It will create many more documents than the current implementation but they will each be significantly smaller than the current documents. Because of this reorganization, any existing Mongo data will not be readable by this PR and will need to be migrated using a future tool.

Warning: Breaking Changes

  • The __init__ signature for Group and Value has changed in this PR.
  • The signature for driver.get(), driver.set(), and driver.clear() has also changed.

@tekulvw tekulvw changed the title [V3 Config] Update Mongo document organization to bypass size restriction [V3 Config] Update Mongo document organization to bypass doc size restriction Mar 31, 2019

tekulvw added some commits Mar 31, 2019

@mikeshardmind

This comment has been minimized.

Copy link
Member

commented Mar 31, 2019

This breaks on at least the all_x methods, as well as on group all methods

From a quick test with a new bot on an isolated server:
image

The only correct result here is the one for past_nicks.

image

I'll look to see what else isn't working as intended with this later.

tekulvw added some commits Mar 31, 2019

@Stonedestroyer

This comment has been minimized.

Copy link
Contributor

commented Mar 31, 2019

As discussed on red discord server apparently we can't use _id due to the fact that it can't be an array. This is an alternative. https://docs.mongodb.com/master/core/index-unique/

It would help making mongo faster than json, i think next step would allowing cog developers to define unique indexes for their own cogs.

tekulvw added some commits Mar 31, 2019

@Tobotimus

This comment has been minimized.

Copy link
Member

commented Apr 1, 2019

@Stonedestroyer You can also simply make _id an object containing a single item with an array as a value, like so:
image

@tekulvw

This comment has been minimized.

Copy link
Member Author

commented Apr 1, 2019

The latest commit changes the layout of the collection/document structure to follow the format in the image below. There will be a single database for the bot, a collection for each combination of "Cog.CATEGORY" and a document for each unique dataset in "Cog.CATEGORY".
image

@mikeshardmind
Copy link
Member

left a comment

This seems ready to go now.

This is a breaking change. This does remove the ability to convert backend right now (which needs to go onto the 3.1 todo list (updating the conversion for the new organization).

@Kowlin

Kowlin approved these changes Apr 3, 2019

@tekulvw tekulvw merged commit 1cd7e41 into Cog-Creators:V3/develop Apr 3, 2019

1 check passed

Travis CI - Pull Request Build Passed
Details
@mikeshardmind mikeshardmind referenced this pull request Apr 5, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.