Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attachment store optimisations #3

Closed
andris9 opened this issue Apr 9, 2017 · 4 comments
Closed

Attachment store optimisations #3

andris9 opened this issue Apr 9, 2017 · 4 comments

Comments

@andris9
Copy link
Member

andris9 commented Apr 9, 2017

Currently attachments stored in GridStore can not be shared between different messages because these contain message specific metadata. The only metadata that should remain should be:

  • messages
  • transferEncoding

When storing new attachments the server should try to find an existing attachment first

database.collection('attachments.files').findOneAndUpdate({
        md5: attachmentMd5,
        length: attachmentLength
    }, {
        $addToSet: {
            messages: messageId
        }
    }, (err, res)=>{
        // if a document was updated (res.value exists) then thats it
        // otherwise create a new attachment instance
    });

This would assume a compound index on md5+length

@andris9
Copy link
Member Author

andris9 commented Apr 9, 2017

Something that needs be solved first are too large indexed arrays. If there's a lot of messages that include the same attachment then the messages array might grow very large

@andris9
Copy link
Member Author

andris9 commented Apr 9, 2017

Removed extra fields from attachment metadata that are not needed with 8c7cedc

@andris9
Copy link
Member Author

andris9 commented Apr 11, 2017

Instead of messages array store referenced message count as the attachment id is stored in mail document anyway

@andris9
Copy link
Member Author

andris9 commented May 16, 2017

Attachments are deduplicated from v1.0.33. Deduplication key is a sha256 hash field. Existing md5 is not used.

@andris9 andris9 closed this as completed May 16, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant