Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CBG-741 - Pre-2.5 _attachment migration #4528

Merged
merged 4 commits into from
Mar 18, 2020
Merged

CBG-741 - Pre-2.5 _attachment migration #4528

merged 4 commits into from
Mar 18, 2020

Conversation

bbrks
Copy link
Member

@bbrks bbrks commented Mar 12, 2020

Performs an in-memory migration of pre-2.5 _attachment metadata into syncData on read before storing in revcache. Does not persist this migration back to the bucket.

New Tests:

  • pre-2.5 attachment, get with rev
  • pre-2.5 attachment, get active
  • pre-2.5 attachment, write noop update
  • pre-2.5 attachment, add new attachment
  • non-conflicting pre-2.5 attachment + 2.5 attachment get active
  • conflicting pre-2.5 attachment + 2.5 attachment get active

Integration Tests

@bbrks bbrks changed the title CBG-741 CBG-741 - Pre-2.5 _attachment migration Mar 12, 2020
@coveralls
Copy link

coveralls commented Mar 13, 2020

Coverage Status

Coverage increased (+0.04%) to 61.843% when pulling 327d3e6 on CBG-741 into 5641409 on master.

@bbrks bbrks marked this pull request as ready for review March 16, 2020 12:10
@bbrks bbrks force-pushed the CBG-741 branch 2 times, most recently from 0e3215e to 59bf30b Compare March 16, 2020 15:58
db/revision_cache_interface.go Outdated Show resolved Hide resolved
db/crud.go Outdated Show resolved Hide resolved
db/crud.go Outdated Show resolved Hide resolved
db/revision_cache_interface.go Outdated Show resolved Hide resolved
adamcfraser and others added 4 commits March 18, 2020 14:28
Return hasPre25Attachments bool from revcache for upgrade case

fix _attachment in test

wip

Remove write part of attachment meta (rely on in-memory moving on each read)

Remove debug

Return error instead of panic

Reorder teardown to prevent panics using walrus

Improve comments, extra assertions on doc syncData

Remove unnecessary changes
…o bodyBytes, add new tests to support mixed pre-2.5 and 2.5 meta
@bbrks
Copy link
Member Author

bbrks commented Mar 18, 2020

Tweaked the approach based on discussion yesterday, and evaluating usages of getRevision to avoid processing _attachments over and over, and doing it in many places.

Black arrows are all calls down into getRevision.
Red underlines are where we're injecting a merged (pre2.5/backup rev "_attachments", and docRev attachments) back in.
Blue arrows are sending merged attachment metadata separately (with cleaned bodies).
Red square is where we're unmarshalling if inline attachments present, and remarshalling a clean body.

IMG_20200318_141819

@bbrks bbrks merged commit e6f8e33 into master Mar 18, 2020
@bbrks bbrks deleted the CBG-741 branch March 18, 2020 16:53
bbrks added a commit that referenced this pull request Mar 18, 2020
)

* Unit test repro

* Update test

Return hasPre25Attachments bool from revcache for upgrade case

fix _attachment in test

wip

Remove write part of attachment meta (rely on in-memory moving on each read)

Remove debug

Return error instead of panic

Reorder teardown to prevent panics using walrus

Improve comments, extra assertions on doc syncData

Remove unnecessary changes

* Reuse inject/MergeAttachment functions when injecting attachments into bodyBytes, add new tests to support mixed pre-2.5 and 2.5 meta

* Update to centralise attachment handling in getRevision

Co-authored-by: Adam Fraser <adam.fraser@couchbase.com>
adamcfraser added a commit that referenced this pull request Mar 18, 2020
) (#4535)

* Unit test repro

* Update test

Return hasPre25Attachments bool from revcache for upgrade case

fix _attachment in test

wip

Remove write part of attachment meta (rely on in-memory moving on each read)

Remove debug

Return error instead of panic

Reorder teardown to prevent panics using walrus

Improve comments, extra assertions on doc syncData

Remove unnecessary changes

* Reuse inject/MergeAttachment functions when injecting attachments into bodyBytes, add new tests to support mixed pre-2.5 and 2.5 meta

* Update to centralise attachment handling in getRevision

Co-authored-by: Adam Fraser <adam.fraser@couchbase.com>

Co-authored-by: Adam Fraser <adam.fraser@couchbase.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants