Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Firestore delete command needs to expose configurable batch size #1185

Closed
samtstern opened this issue Apr 8, 2019 · 2 comments
Closed

Firestore delete command needs to expose configurable batch size #1185

samtstern opened this issue Apr 8, 2019 · 2 comments

Comments

@samtstern
Copy link
Contributor

[REQUIRED] Environment info

All versions, all platforms.

[REQUIRED] Test case

See below.

[REQUIRED] Steps to reproduce

Right now we delete documents in batches of 250:
https://github.com/firebase/firebase-tools/blob/master/src/firestore/delete.js#L279

Since we only have the document ID at the time of deletion, we have no idea how large the documents are or how many indexes they belong to.

Firestore transactions/batched writes have two limits:

  • Max number of index writes/deletes (I think it's 10,000)
  • Max total size (I think it's 10MB)

So in issues like bug 129094738 the documents are large or have many indexes which makes them impossible to delete.

[REQUIRED] Expected behavior

We need two things here:

  • To catch these 400 errors and make it clear what happened.
  • Expose a --batchSize or some other parameter to do fewer deletes at once, allowing users to get around this error by lowering the limit (trial and error)

[REQUIRED] Actual behavior

400 errors being returned to clients with no explanation.

@mikelehen
Copy link
Contributor

It might be neat if it could automatically half the batch size each time it gets a 400 and retry... Just as the tool can't know if the documents are going to be too big, the user probably won't know either... And there might only be a small number of documents that are problematic, meaning it's undesirable to reduce the batch size for the entire job...

@samtstern
Copy link
Contributor Author

I believe this was fixed by #1331 but never marked as such,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants