Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[V3 Config] Update Mongo document organization to bypass doc size restriction #2536

Merged
merged 16 commits into from Apr 3, 2019

Conversation

@tekulvw
Copy link
Member

@tekulvw tekulvw commented Mar 31, 2019

Type

  • Bugfix
  • Enhancement
  • New feature

Description of the changes

This PR reorganizes how the Mongo driver stores data on the database. It will create many more documents than the current implementation but they will each be significantly smaller than the current documents. Because of this reorganization, any existing Mongo data will not be readable by this PR and will need to be migrated using a future tool.

Warning: Breaking Changes

  • The __init__ signature for Group and Value has changed in this PR.
  • The signature for driver.get(), driver.set(), and driver.clear() has also changed.
@tekulvw tekulvw changed the title [V3 Config] Update Mongo document organization to bypass size restriction [V3 Config] Update Mongo document organization to bypass doc size restriction Mar 31, 2019
@mikeshardmind
Copy link
Contributor

@mikeshardmind mikeshardmind commented Mar 31, 2019

This breaks on at least the all_x methods, as well as on group all methods

From a quick test with a new bot on an isolated server:
image

The only correct result here is the one for past_nicks.

image

I'll look to see what else isn't working as intended with this later.

Loading

@Stonedestroyer
Copy link
Contributor

@Stonedestroyer Stonedestroyer commented Mar 31, 2019

As discussed on red discord server apparently we can't use _id due to the fact that it can't be an array. This is an alternative. https://docs.mongodb.com/master/core/index-unique/

It would help making mongo faster than json, i think next step would allowing cog developers to define unique indexes for their own cogs.

Loading

@Tobotimus
Copy link
Member

@Tobotimus Tobotimus commented Apr 1, 2019

@Stonedestroyer You can also simply make _id an object containing a single item with an array as a value, like so:
image

Loading

@tekulvw
Copy link
Member Author

@tekulvw tekulvw commented Apr 1, 2019

The latest commit changes the layout of the collection/document structure to follow the format in the image below. There will be a single database for the bot, a collection for each combination of "Cog.CATEGORY" and a document for each unique dataset in "Cog.CATEGORY".
image

Loading

Copy link
Contributor

@mikeshardmind mikeshardmind left a comment

This seems ready to go now.

This is a breaking change. This does remove the ability to convert backend right now (which needs to go onto the 3.1 todo list (updating the conversion for the new organization).

Loading

Kowlin
Kowlin approved these changes Apr 3, 2019
@tekulvw tekulvw merged commit 1cd7e41 into Cog-Creators:V3/develop Apr 3, 2019
1 check passed
Loading
@jack1142 jack1142 added this to the 3.1.0 milestone Nov 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants