Firestore "400, Request contains an invalid argument" error when uploading large rules file (55.2KB)

[brianneisler]

[REQUIRED] Environment info

firebase-tools: 8.13.1

Platform: macOS Catalina 10.15.7

[REQUIRED] Test case

When attempting to deploy a larger rules file (currently 55.2KB) the rules fail to deploy and I receive a "400, Request contains an invalid argument" error.

I've created a minimal project that reproduces the issue here https://github.com/brianneisler/firestore-large-rules-file

This project contains a simple firebase project which only has firestore configured. The rules file that is contained in this project has been verified by Firebase support to not contain any bugs or issues as well as tested with firebase emulator which outputs no errors or warnings when loading the rules. It is simply a larger set of rules from an open source firebase application framework I've been working on called moltres. When attempting to deploy this project to firebase, a 400 error from firebase is returned.

HTTP Error: 400, Request contains an invalid argument.

When attempting to manually copy and paste the rules to the rules editor and then save the rules an error occurs.

Error saving rules - an unknown error occurred

I ran through a test where I commented out all of the code and then slowly uncommented portions of it based on dependency (looking for a problematic line). However, what I noticed is that as I uncommented more and more code the problem started happening more frequently. At first it would fail 1 in 5 times. Uncomment more code and it started failing more frequently. The rules contained in this project are at the threshold where they contain enough code that it does not publish most of the time. I was able to get them to publish once but only after clicking the publish button 20 or so times. This leads me to believe that there is some kind of race condition based on compilation time on Firebase's side since the size of the file remained the same, only the code that was uncommented changed.

I was very excited when Firebase announced that they had lifted the max rules size to 256KB as my projects usually run on the larger size and I was continually bumping into the upper limit. I even created a rules minifier to work around the problem, firemin. However, it seems that whatever changes were made to the compiler actually made the issue worse.

I have verified both personally and with firebase support that this rules file has no errors or warnings in it.

I submitted this issue to Firebase support and they were unable to help me. Their only suggestion was to try uploading the rules in chunks (pasting half of it first, saving, and then pasting the other half.) This didn't seem to change anything.

I also submitted this issue to StackOverflow and received no helpful responses https://stackoverflow.com/questions/63925021/firestore-uploading-larger-rules-file-results-in-error-400-request-contains-a?noredirect=1#comment113059218_63925021

[REQUIRED] Steps to reproduce

Reproduction Setup

• create a new project on firebase.google.com using the "add new project"
• go to firestore tab and click "Create database"
• select "start in production mode" and click "Next"
• select nam-5 (us-central) and click "Enable"

Reproduction on firebase console

• click "rules" tab
• copy rules from the repo https://github.com/brianneisler/firestore-large-rules-file/blob/main/firestore.rules and paste into editor
• click "publish" button
• will likely come back with "Error saving rules - an unknown error occurred"
• note here that if i clicked "publish" enough times occasionally it will save.

Reproduction locally from example repo

• checkout the example repo https://github.com/brianneisler/firestore-large-rules-file
• on command line, install project dependencies npm i
• replace the project name in .firebaserc with the project name you used during setup
• publish rules only ./node_modules/.bin/firebase deploy --only firestore:rules --debug
• An error should be returned and rules will fail to deploy. Error will look
  like this...

Error: HTTP Error: 400, Request contains an invalid argument.
[2020-09-13T19:29:41.552Z] Error Context: {
  "body": {
    "error": {
      "code": 400,
      "message": "Request contains an invalid argument.",
      "status": "INVALID_ARGUMENT"
    }
  },
  "response": {
    "statusCode": 400,
    "body": {
      "error": {
        "code": 400,
        "message": "Request contains an invalid argument.",
        "status": "INVALID_ARGUMENT"
      }
    },
    "headers": {
      "vary": "X-Origin, Referer, Origin,Accept-Encoding",
      "content-type": "application/json; charset=UTF-8",
      "date": "Sun, 13 Sep 2020 19:29:41 GMT",
      "server": "ESF",
      "cache-control": "private",
      "x-xss-protection": "0",
      "x-frame-options": "SAMEORIGIN",
      "x-content-type-options": "nosniff",
      "alt-svc": "h3-29=\":443\"; ma=2592000,h3-27=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-T050=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"",
      "accept-ranges": "none",
      "transfer-encoding": "chunked"
    },
    "request": {
      "uri": {
        "protocol": "https:",
        "slashes": true,
        "auth": null,
        "host": "firebaserules.googleapis.com",
        "port": 443,
        "hostname": "firebaserules.googleapis.com",
        "hash": null,
        "search": null,
        "query": null,
        "pathname": "/v1/projects/brianneisler-test:test",
        "path": "/v1/projects/brianneisler-test:test",
        "href": "https://firebaserules.googleapis.com/v1/projects/brianneisler-test:test"
      },
      "method": "POST"
    }
  }
}

Reproduction locally using fresh project

• on command line, init for existing project
• npm i -g firebase-tools
• firebase init
• using spacebar select "firestore"
• select "use an existing project"
• select the name of the existing project created during setup
• click enter
• accept the default file names for firestore.rules and firestore.indexes.json
• replace the firestore.rules file with the file from example project
• publish rules only firebase deploy --only firestore:rules --debug
• An error should be returned and rules will faile to deploy. Error will look
  like this...

Error: HTTP Error: 400, Request contains an invalid argument.
[2020-09-13T19:29:41.552Z] Error Context: {
  "body": {
    "error": {
      "code": 400,
      "message": "Request contains an invalid argument.",
      "status": "INVALID_ARGUMENT"
    }
  },
  "response": {
    "statusCode": 400,
    "body": {
      "error": {
        "code": 400,
        "message": "Request contains an invalid argument.",
        "status": "INVALID_ARGUMENT"
      }
    },
    "headers": {
      "vary": "X-Origin, Referer, Origin,Accept-Encoding",
      "content-type": "application/json; charset=UTF-8",
      "date": "Sun, 13 Sep 2020 19:29:41 GMT",
      "server": "ESF",
      "cache-control": "private",
      "x-xss-protection": "0",
      "x-frame-options": "SAMEORIGIN",
      "x-content-type-options": "nosniff",
      "alt-svc": "h3-29=\":443\"; ma=2592000,h3-27=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-T050=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"",
      "accept-ranges": "none",
      "transfer-encoding": "chunked"
    },
    "request": {
      "uri": {
        "protocol": "https:",
        "slashes": true,
        "auth": null,
        "host": "firebaserules.googleapis.com",
        "port": 443,
        "hostname": "firebaserules.googleapis.com",
        "hash": null,
        "search": null,
        "query": null,
        "pathname": "/v1/projects/brianneisler-test:test",
        "path": "/v1/projects/brianneisler-test:test",
        "href": "https://firebaserules.googleapis.com/v1/projects/brianneisler-test:test"
      },
      "method": "POST"
    }
  }
}

Current workaround

If I minimize the rules using the firemin rules minifier, the deployment succeeds more often. This makes sense since it collapses single use functions and overall reduces the size of the rules file. You can test this by trying to upload the firestore.min.rules file contained in the project

[REQUIRED] Expected behavior

The rules file should deploy without error.

[REQUIRED] Actual behavior

Receive a "400, Request contains an invalid argument" error

[samtstern]

@brianneisler thank you for the super detailed report (and also for making firemin, which we loved). I'm going to have to circulate this internally and get some people on the rules backend to take a look, will update here when I have news.

[samtstern]

For Googlers: this is b/171547886 internally

[yaoonline]

Hi! How did you solve the problem? I have the same (((

[brianneisler]

@yaoonline currently still unresolved. The best workarounds i've found so far are to...

1. Use the firemin rules minifier to reduce the number of statements in your rules file and the overall size of the file.
2. Use a gigabit ethernet line when trying to deploy your rules. The faster connection seems to ease the issue on whatever internal timeout on the Google side is causing this error.
3. Setup your deployment in such a way where the rules are independently deployed from the actual app deployment. This way you can setup retry mechanisms to attempt to upload the rules several times before finally giving up. It definitely seems that whatever is causing the issue is race condition related on the Google side, so often the rules upload will succeed with enough attempts.

What I haven't tried yet is setting up a cloud run script to perform the actual deployment. This way the latency between the the deployment code and the google infrastructure would be significantly reduced.

Anyways, hope that helps! let me know if you figure anything else out as this issue has been a real pain point for releases.

Also, please upvote as the only way this is going to get any attention on the Google side is if enough people report having this issue. I think apps with a large rules set are a very small minority within the Firebase developer community. Seems like most people are building their app's rules with some read restrictions but routing all writes over an http app which results in a much smaller rules file since you don't have to include any write rules.

[brianneisler]

@samtstern any word on this one? I'm at the point where my rules file can no longer be deployed and it's preventing further development.

[SamyPesse]

Thank you @brianneisler for firemin! In our case it was not possible to use it because of some firestore syntax, here is a quick alternative with regex (less reliable): https://gist.github.com/SamyPesse/d88a44bafba6752317fdbc74848f4242
It helped us go from 120KB to 54KB which improved the reliability of deployment but it's still a major pain.
We had to setup some retry logic to mitigate it.

[brianneisler]

hey @SamyPesse, let me know what prevented you from being able to use firemin and i'll take a look at supporting the functionality. I haven't added function short names yet but it would be pretty easy to implement.

[brianneisler]

@SamyPesse, thanks for reporting the issue with firemin. It's now been resolved with release v0.3.1 brianneisler/firemin#36

[dteirney]

@samtstern is there any update on when the flakiness of the firestore rules deploy might be addressed? We've started running into what we assume is this issue. Finding this ticket was somewhat surprising after needing to turn on --debug to figure out why our CI deploys were frequently failing.

[samtstern]

@dteirney let me try to escalate this again

[samtstern]

Ok I have raised the priority of this internally, the US offices are out for the day but I'll make sure someone takes a look at this again later this week. For reference the internal Bug ID is b/168860909, if anyone here files a support case about this issue make sure to mention that ID.

@dteirney I will follow up on this however I am going to close the issue here since there's no possible fix in the CLI, this is a backend rules deployment issue, hope that makes sense!

[crobinson42]

This error also occurs when trying to send a push-notification to an iOS simulator device.