This repository was archived by the owner on Mar 2, 2026. It is now read-only.

feat: add bulk writer#396

Merged

craiglabenz merged 5 commits into

bulk-writer-release

Aug 11, 2021

craiglabenz commented Jul 15, 2021 •

edited

Loading

Contributor

Adds the new BulkWriter feature, as assembled from these PRs:

craiglabenz added the do not merge label

craiglabenz requested review from a team

July 15, 2021 17:13

product-auto-label Bot added the api: firestore label

google-cla Bot commented Jul 15, 2021

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla Bot added the cla: no label

tseaver commented Jul 15, 2021

Contributor

@googlebot I consent.

google-cla Bot commented Jul 15, 2021

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

1 similar comment

google-cla Bot commented Jul 22, 2021

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

tseaver commented Jul 22, 2021

Contributor

@googlebot I consent.

google-cla Bot commented Jul 22, 2021

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

tseaver commented Jul 22, 2021

Contributor

@googlebot I fixed it.

google-cla Bot commented Jul 22, 2021

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

tseaver commented Jul 22, 2021

Contributor

@craiglabenz I give up on trying to appease @googlebot.

google-cla Bot commented Jul 22, 2021

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

1 similar comment

google-cla Bot commented Aug 2, 2021

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

craiglabenz and others added 3 commits

August 10, 2021 14:02


          feat: bulk writer 555 rate_limiter (#368)

5a0ad83

* added 555 throttle utility

* Update google/cloud/firestore_v1/throttle.py

Co-authored-by: Tres Seaver <tseaver@palladion.com>

* added ability to request a number of tokens

* replaced Callable now parameter with module function

* updated tests

* renamed throttle -> ramp up

* improved docstrings

* linting

* fixed test coverage

* rename to RateLimiter and defer clock to first op

* linting

Co-authored-by: Tres Seaver <tseaver@palladion.com>


          feat: added new batch class for BulkWriter (#397)

783d9e3

* feat: added new batch class for BulkWriter

* updated docstring to use less colloquial language


          feat: BulkWriter implementation (#384)

70e255d

* feat: added `write` method to batch classes

* added docstrings to all 3 batch classes

instead of just the base

* updated batch classes to remove control flag

now branches logic via subclasses

* fixed broken tests off abstract class

* fixed docstring

* refactored BulkWriteBatch

this commit increases the distance between WriteBatch and BulkWriteBatch

* began adding [Async]BulkWriter

* continued implementation

* working impl or BW

* tidied up BW impl

* beginning of unit tests for BW

* fixed merge problem

* initial set of BW unit tests

* refactored bulkwriter sending mechanism

now consumes off the queue and schedules on the main thread, only going async to actually send

* final CI touch ups

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md

* moved BulkWriter parameters to options format

* rebased off master

* test fixes

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

craiglabenz force-pushed the bulk-writer-release branch from a25b169 to 70e255d Compare

August 10, 2021 18:02

google-cla Bot commented Aug 10, 2021

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.


          feat: add retry support for BulkWriter errors (#413)

510f15b

* parent 0176cc7
author Craig Labenz <craig.labenz@gmail.com> 1623693904 -0700
committer Craig Labenz <craig.labenz@gmail.com> 1628617523 -0400

feat: add retries to bulk-writer

* fixed rebase error

google-cla Bot commented Aug 10, 2021

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

craiglabenz removed the do not merge label


          Merge branch 'master' into bulk-writer-release

8956f42

google-cla Bot commented Aug 11, 2021

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

craiglabenz changed the title ~~feat: bulk writer [WIP]~~ feat: add bulk writer

crwilcox commented Aug 11, 2021

Contributor

Code has been previously reviewed 👍

crwilcox added cla: yes and removed cla: no labels

crwilcox approved these changes

View reviewed changes

craiglabenz merged commit 98a7753 into master

craiglabenz deleted the bulk-writer-release branch

August 11, 2021 16:41

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

api: firestore cla: yes