Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Database emulator does not support .indexOn #1980

Closed
filipesilva opened this issue Feb 18, 2020 · 22 comments · Fixed by #2043
Closed

Database emulator does not support .indexOn #1980

filipesilva opened this issue Feb 18, 2020 · 22 comments · Fixed by #2043

Comments

@filipesilva
Copy link

[REQUIRED] Environment info

firebase-tools: 3.3.0

Platform: Windows

[REQUIRED] Test case

I used the official docs example for orderByChild https://firebase.google.com/docs/database/security/indexing-data?authuser=0.

// client code
      db.ref('dinosaurs').set({
        "lambeosaurus": {
          "height": 2.1,
          "length": 12.5,
          "weight": 5000
        },
        "stegosaurus": {
          "height": 4,
          "length": 9,
          "weight": 2500
        }
      }).then(()=>{
        db.ref('dinosaurs').orderByChild('height').startAt(3).once('value').then(s => console.log(s.val()));
      })
// security rules
{
  "rules": {
    ".read": true,
    ".write": true,
    "dinosaurs": {
      ".indexOn": [
        "height",
        "length"
      ]
    }
  }
}

[REQUIRED] Steps to reproduce

Running the code above against a real database shows stegosaurus: {height: 4, length: 9, weight: 2500}.

Running it against the emulator shows both stegosaurus: {height: 4, length: 9, weight: 2500} and a warning:

logger.ts:86 [2020-02-18T17:08:23.851Z]  @firebase/database: FIREBASE WARNING: Using an unspecified index. Your data will be downloaded and filtered on the client. Consider adding ".indexOn": "height" at /dinosaurs to your security rules for better performance. 

[REQUIRED] Expected behavior

I expected .indexOn to be supported, or perhaps for the emulator to mention it was not using the index, or for this limitation to be listed somewhere. Maybe it's already documented but I couldn't find it in the official docs or the issue tracker.

[REQUIRED] Actual behavior

The emulator silently ignores the index and the client warns that there is no index.

@bkendall
Copy link
Contributor

That's a suuuper old version of firebase-tools. Can you run firebase --version to verify it?

@filipesilva
Copy link
Author

Heya @bkendall, here's the output:

kamik@RED-X1C6 MINGW64 /d/sandbox/firebase-emulator (master)
$ yarn firebase --version
yarn run v1.21.1
$ D:\sandbox\firebase-emulator\node_modules\.bin\firebase --version
7.11.0

@filipesilva
Copy link
Author

Sorry, now I see that I mixed up the version of firebase-functions with firebase-tools!!

@filipesilva
Copy link
Author

Also tried with 7.13.1, same result.

@yuchenshi
Copy link
Member

Could you please check if the rules are properly loaded in the the emulator by curl localhost:9000/.settings/rules.json?ns=your-database-name -H 'Authorization: Bearer owner'? (Replace localhost:9000 with your emulator host/port and your-database-name with your database name).

Also, please double check that you're connecting to the same database in your web app as specified to Firebase CLI. On the web app, make sure you're specifying the db name like databaseURL: "http://localhost:9000/?ns=foo" (note the ns parameter). And when starting Firebase CLI, make sure to use firebase emulators:start --project foo, because Firebase CLI automatically set the security rules only for the database whose name matches the project ID. If you are accessing a different database, that database will have the default open security rules and no index entries.

@yuchenshi yuchenshi added the Needs: Author Feedback Issues awaiting author feedback label Feb 18, 2020
@filipesilva
Copy link
Author

@yuchenshi thanks for giving me all the important things to check. I've made a repro that you can hopefully use to see the same results as I'm seeing, and I think I've also been able to narrow down the nature of the problem.

To repro follow these steps:

  • first clone and start the project
git clone https://github.com/filipesilva/firebase-emulator-orderby-index
cd firebase-emulator-orderby-index
yarn emulators
{stegosaurus: {…}}
logger.ts:86 [2020-02-18T20:37:11.763Z]  @firebase/database: FIREBASE WARNING: Using an unspecified index. Your data will be downloaded and filtered on the client. Consider adding ".indexOn": "height" at /dinosaurs to your security rules for better performance. 
  • you can edit public/index.html and flip var useEmulators = true; to false to connect to a real database, where the warning won't be present
  • at this point you can run curl localhost:9000/.settings/rules.json?ns=emulator-test-1 -H 'Authorization: Bearer owner' as well
kamik@RED-X1C6 MINGW64 /d/sandbox
$ curl localhost:9000/.settings/rules.json?ns=emulator-test-1 -H 'Authorization: Bearer owner'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    70  100    70    0     0    280      0 --:--:-- --:--:-- --:--:--   280{
    "rules": {
        ".read": true,
        ".write": true
    }
}
  • this result surprised me, because I can see the rules on disk and they don't look like this
  • if you save the rules, even without any change, you'll see this on the emulator log
i  database: Change detected, updating rules...
+  database: Rules updated.
  • running curl again shows the right rules now
kamik@RED-X1C6 MINGW64 /d/sandbox
$ curl localhost:9000/.settings/rules.json?ns=emulator-test-1 -H 'Authorization: Bearer owner'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   147  100   147    0     0    671      0 --:--:-- --:--:-- --:--:--   671{
  "rules": {
    ".read": true,
    ".write": true,
    "dinosaurs": {
      ".indexOn": [
        "height",
        "length"
      ]
    }
  }
}
  • if you try to save the rules again, you'll see this:
i  database: Change detected, updating rules...
!  database.rules.json:No data supplied.
!  Failed to update rules
  • changes in the rules file won't be shown in curl anymore

So it seems that the problem here is actually related to loading the rules file. It isn't loaded initially, but will be loaded on the first save. Any save after the first will fail to load.

@google-oss-bot google-oss-bot added Needs: Attention and removed Needs: Author Feedback Issues awaiting author feedback labels Feb 18, 2020
@yuchenshi
Copy link
Member

@samtstern Could you please take a look at the repro when you get some time since you are most familiar with the rules (re)loading part?

@yuchenshi yuchenshi assigned samtstern and unassigned yuchenshi Feb 18, 2020
@samtstern
Copy link
Contributor

Hmm so this is a little strange.

When I tried these rules the first time:

{
  "rules": {
    ".read": "auth != null",
    ".write": "auth != null",
    "dinosaurs": {
      ".indexOn": [
        "height",
        "length"
      ]
    }
  }
}

I got the error mentioned above:

i  database: Change detected, updating rules...
⚠  database.rules.json:No data supplied.
⚠  Failed to update rules

Then I went back to just read/write=true rules and got success. Then I tried the "bad" rules on more time and also got success:

i  database: Change detected, updating rules...
✔  database: Rules updated

And now I can't get the same failure as before to repeat at all. I think I need someone to help me investigate where the "no data supplied" error comes from.

@samtstern
Copy link
Contributor

@IanWyszynski could you look into the RTDB emulator source and see why we'd get No data supplied?

@jmwski
Copy link
Contributor

jmwski commented Feb 21, 2020

@IanWyszynski could you look into the RTDB emulator source and see why we'd get No data supplied?

@samstern In the ./settings/rules.json serving path, the only way the emulator returns that error is if the body of the PUT request is empty. Did you log:

const newContent = fs.readFileSync(rulesPath).toString();

to check that we're not accidentally sending an empty rules payload?

@samtstern
Copy link
Contributor

@IanWyszynski thanks! Yeah if I replace newContent with undefined I do get the "No data supplied" error however that's the only way I can reproduce this now ... I can no longer get it to happen just by putting in certain rules (that only happened once).

@filipesilva is this still happening consistently for you?

@filipesilva
Copy link
Author

@samtstern tried running yarn emulators followed by saving the database.rules.json files a couple of times without any changes, and repeated the process.

kamik@RED-X1C6 MINGW64 /d/sandbox/firebase-emulator-orderby-index (master)
$ yarn emulators
yarn run v1.21.1
$ yarn firebase emulators:start --project emulator-test-1
$ D:\sandbox\firebase-emulator-orderby-index\node_modules\.bin\firebase emulators:start --project emulator-test-1
i  emulators: Starting emulators: database, hosting
i  database: Emulator logging to database-debug.log
+  database: Emulator started at http://localhost:9002
i  database: For testing set FIREBASE_DATABASE_EMULATOR_HOST=localhost:9002
i  hosting: Serving hosting files from: public
+  hosting: Local server: http://localhost:5002
+  hosting: Emulator started at http://localhost:5002
+  All emulators started, it is now safe to connect.
i  database: Change detected, updating rules...
+  database: Rules updated.
i  database: Change detected, updating rules...
!  database.rules.json:No data supplied.
!  Failed to update rules
i  database: Change detected, updating rules...
!  database.rules.json:No data supplied.
!  Failed to update rules
i  Shutting down emulators.
i  Stopping database emulator

kamik@RED-X1C6 MINGW64 /d/sandbox/firebase-emulator-orderby-index (master)
$ yarn emulators
yarn run v1.21.1
$ yarn firebase emulators:start --project emulator-test-1
$ D:\sandbox\firebase-emulator-orderby-index\node_modules\.bin\firebase emulators:start --project emulator-test-1
i  emulators: Starting emulators: database, hosting
i  database: Emulator logging to database-debug.log
+  database: Emulator started at http://localhost:9002
i  database: For testing set FIREBASE_DATABASE_EMULATOR_HOST=localhost:9002
i  hosting: Serving hosting files from: public
+  hosting: Local server: http://localhost:5002
+  hosting: Emulator started at http://localhost:5002
+  All emulators started, it is now safe to connect.
i  database: Change detected, updating rules...
+  database: Rules updated.
i  database: Change detected, updating rules...
+  database: Rules updated.
i  database: Change detected, updating rules...
!  database.rules.json:No data supplied.
!  Failed to update rules
i  database: Change detected, updating rules...
!  database.rules.json:No data supplied.
!  Failed to update rules
i  database: Change detected, updating rules...
!  database.rules.json:No data supplied.
!  Failed to update rules
i  Shutting down emulators.
i  Stopping database emulator
!  database emulator has exited upon receiving signal: SIGINT
i  Stopping hosting emulator

kamik@RED-X1C6 MINGW64 /d/sandbox/firebase-emulator-orderby-index (master)
$ yarn emulators
yarn run v1.21.1
$ yarn firebase emulators:start --project emulator-test-1
$ D:\sandbox\firebase-emulator-orderby-index\node_modules\.bin\firebase emulators:start --project emulator-test-1
i  emulators: Starting emulators: database, hosting
i  database: Emulator logging to database-debug.log
+  database: Emulator started at http://localhost:9002
i  database: For testing set FIREBASE_DATABASE_EMULATOR_HOST=localhost:9002
i  hosting: Serving hosting files from: public
+  hosting: Local server: http://localhost:5002
+  hosting: Emulator started at http://localhost:5002
+  All emulators started, it is now safe to connect.
i  database: Change detected, updating rules...
+  database: Rules updated.
i  database: Change detected, updating rules...
+  database: Rules updated.
i  database: Change detected, updating rules...
!  database.rules.json:No data supplied.
!  Failed to update rules
i  database: Change detected, updating rules...
!  database.rules.json:No data supplied.
!  Failed to update rules
i  database: Change detected, updating rules...
!  database.rules.json:No data supplied.
!  Failed to update rules
i  Shutting down emulators.
i  Stopping database emulator
!  database emulator has exited upon receiving signal: SIGINT
i  Stopping hosting emulator

kamik@RED-X1C6 MINGW64 /d/sandbox/firebase-emulator-orderby-index (master)
$ yarn emulators
yarn run v1.21.1
$ yarn firebase emulators:start --project emulator-test-1
$ D:\sandbox\firebase-emulator-orderby-index\node_modules\.bin\firebase emulators:start --project emulator-test-1
i  emulators: Starting emulators: database, hosting
i  database: Emulator logging to database-debug.log
+  database: Emulator started at http://localhost:9002
i  database: For testing set FIREBASE_DATABASE_EMULATOR_HOST=localhost:9002
i  hosting: Serving hosting files from: public
+  hosting: Local server: http://localhost:5002
+  hosting: Emulator started at http://localhost:5002
+  All emulators started, it is now safe to connect.
i  database: Change detected, updating rules...
+  database: Rules updated.
i  database: Change detected, updating rules...
!  database.rules.json:No data supplied.
!  Failed to update rules
i  database: Change detected, updating rules...
!  database.rules.json:No data supplied.
!  Failed to update rules
i  database: Change detected, updating rules...
!  database.rules.json:No data supplied.
!  Failed to update rules
i  Shutting down emulators.
i  Stopping database emulator

I could always reproduce, but sometimes it took two saves to get database.rules.json:No data supplied..

I wonder if the OS is a factor. I'm on Windows 10 (build 1904184), using node 12.4.0, and yarn 1.21.1.

@samtstern
Copy link
Contributor

Thanks for the logs! We clearly have insufficient logging here so I am adding #1991 to address that at least. It does seem like some of our filesystem reads are failing but I don't know why ...

@filipesilva
Copy link
Author

@samtstern I updated to firebase-tools@7.14.0, which I think includes the extra logging in #1991 and followed the same repro above (start emulators, save the database rules with no change a few times):

kamik@RED-X1C6 MINGW64 /d/sandbox/firebase-emulator-orderby-index (master)
$ yarn firebase emulators:start --project emulator-test-1
yarn run v1.21.1
$ D:\sandbox\firebase-emulator-orderby-index\node_modules\.bin\firebase emulators:start --project emulator-test-1
i  emulators: Starting emulators: database, hosting
+  hub: emulator hub started at http://localhost:4400
i  database: downloading firebase-database-emulator-v4.4.0.jar...
Progress: ============================================================================================================================================> (100% of 28MB) 
i  database: Removing outdated emulator files: firebase-database-emulator-v4.3.1.jar
i  database: database emulator logging to database-debug.log
+  database: database emulator started at http://localhost:9002
i  database: For testing set FIREBASE_DATABASE_EMULATOR_HOST=localhost:9002
i  hosting: Serving hosting files from: public
+  hosting: Local server: http://localhost:5002
+  hosting: hosting emulator started at http://localhost:5002
+  All emulators started, it is now safe to connect.
i  database: Change detected, updating rules...
+  database: Rules updated.
i  database: Change detected, updating rules...
!  database.rules.json:No data supplied.
!  Failed to update rules
i  database: Change detected, updating rules...
!  database.rules.json:No data supplied.
!  Failed to update rules

Also tried it with the debug flag:

kamik@RED-X1C6 MINGW64 /d/sandbox/firebase-emulator-orderby-index (master)
$ yarn firebase emulators:start --project emulator-test-1 --debug
yarn run v1.21.1
$ D:\sandbox\firebase-emulator-orderby-index\node_modules\.bin\firebase emulators:start --project emulator-test-1 --debug
[2020-03-02T11:09:44.240Z] ----------------------------------------------------------------------
[2020-03-02T11:09:44.243Z] Command:       C:\Program Files\nodejs\node.exe D:\sandbox\firebase-emulator-orderby-index\node_modules\firebase-tools\lib\bin\firebase.js e
mulators:start --project emulator-test-1 --debug
[2020-03-02T11:09:44.244Z] CLI Version:   7.14.0
[2020-03-02T11:09:44.244Z] Platform:      win32
[2020-03-02T11:09:44.244Z] Node Version:  v12.4.0
[2020-03-02T11:09:44.245Z] Time:          Mon Mar 02 2020 11:09:44 GMT+0000 (Greenwich Mean Time)
[2020-03-02T11:09:44.246Z] ----------------------------------------------------------------------
[2020-03-02T11:09:44.246Z]
[2020-03-02T11:09:44.254Z] > command requires scopes: ["email","openid","https://www.googleapis.com/auth/cloudplatformprojects.readonly","https://www.googleapis.com/au
th/firebase","https://www.googleapis.com/auth/cloud-platform"]
[2020-03-02T11:09:44.254Z] > authorizing via signed-in user
i  emulators: Starting emulators: database, hosting
[2020-03-02T11:09:46.338Z] [hub] writing locator at C:\Users\kamik\AppData\Local\Temp\hub-emulator-test-1.json
+  hub: emulator hub started at http://localhost:4400
[2020-03-02T11:09:48.388Z] Ignoring unsupported arg: projectId
[2020-03-02T11:09:48.389Z] Ignoring unsupported arg: auto_download
[2020-03-02T11:09:48.389Z] Ignoring unsupported arg: rules
[2020-03-02T11:09:48.389Z] Starting database emulator with command {"binary":"java","args":["-Duser.language=en","-jar","C:\\Users\\kamik\\.cache\\firebase\\emulators\
\firebase-database-emulator-v4.4.0.jar","--host","localhost","--port",9002],"optionalArgs":["port","host","functions_emulator_port","functions_emulator_host"],"joinArg
s":false}
i  database: database emulator logging to database-debug.log
[2020-03-02T11:09:49.994Z] WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/C:/Users/kamik/.cache/firebase/emulators/firebase-database-emulator-v4.4.0.jar) to f
ield sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

[2020-03-02T11:09:50.705Z] 11:09:50.702 [NamespaceSystem-akka.actor.default-dispatcher-4] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started

[2020-03-02T11:09:50.867Z] 11:09:50.865 [main] INFO com.firebase.server.forge.App$ - Listening at localhost:9002

+  database: database emulator started at http://localhost:9002
i  database: For testing set FIREBASE_DATABASE_EMULATOR_HOST=localhost:9002
[2020-03-02T11:09:53.698Z] >>> HTTP REQUEST GET https://firebase.googleapis.com/v1beta1/projects/emulator-test-1/webApps/-/config

[2020-03-02T11:09:54.437Z] <<< HTTP RESPONSE 200 content-type=application/json; charset=UTF-8, vary=X-Origin, Referer, Origin,Accept-Encoding, date=Mon, 02 Mar 2020 11
:09:54 GMT, server=ESF, cache-control=private, x-xss-protection=0, x-frame-options=SAMEORIGIN, x-content-type-options=nosniff, alt-svc=quic=":443"; ma=2592000; v="46,4
3",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000, accept-ranges=none, transfer
-encoding=chunked
i  hosting: Serving hosting files from: public
+  hosting: Local server: http://localhost:5002
+  hosting: hosting emulator started at http://localhost:5002
+  All emulators started, it is now safe to connect.
i  database: Change detected, updating rules...
[2020-03-02T11:09:57.032Z] >>> HTTP REQUEST PUT http://localhost:9002/.settings/rules.json?ns=emulator-test-1
 {
  "rules": {
    ".read": true,
    ".write": true,
    "dinosaurs": {
      ".indexOn": [
        "height",
        "length"
      ]
    }
  }
}
[2020-03-02T11:09:57.390Z] <<< HTTP RESPONSE 200 content-length=15, content-type=application/json; charset=utf-8, access-control-allow-origin=*, cache-control=no-cache
, x-firebase-project-id=emulator-test-1, x-firebase-project-number=123456789, x-firebase-uuid=7bbc1cd3-3611-4a4a-b866-ea7f56b3ff1a
+  database: Rules updated.
i  database: Change detected, updating rules...
[2020-03-02T11:09:59.724Z] >>> HTTP REQUEST PUT http://localhost:9002/.settings/rules.json?ns=emulator-test-1

[2020-03-02T11:09:59.733Z] <<< HTTP RESPONSE 400 content-length=38, content-type=application/json; charset=utf-8, access-control-allow-origin=*, cache-control=no-cache
, x-firebase-project-id=emulator-test-1, x-firebase-project-number=123456789, x-firebase-uuid=7bbc1cd3-3611-4a4a-b866-ea7f56b3ff1a
[2020-03-02T11:09:59.734Z] <<< HTTP RESPONSE BODY {
  "error" : "No data supplied."
}

!  database.rules.json:No data supplied.
!  Failed to update rules
i  database: Change detected, updating rules...
[2020-03-02T11:10:03.484Z] >>> HTTP REQUEST PUT http://localhost:9002/.settings/rules.json?ns=emulator-test-1
 {
  "rules": {
    ".read": true,
    ".write": true,
    "dinosaurs": {
      ".indexOn": [
        "height",
        "length"
      ]
    }
  }
}
[2020-03-02T11:10:03.494Z] <<< HTTP RESPONSE 200 content-length=15, content-type=application/json; charset=utf-8, access-control-allow-origin=*, cache-control=no-cache
, x-firebase-project-id=emulator-test-1, x-firebase-project-number=123456789, x-firebase-uuid=7bbc1cd3-3611-4a4a-b866-ea7f56b3ff1a
+  database: Rules updated.
i  database: Change detected, updating rules...
[2020-03-02T11:10:05.987Z] >>> HTTP REQUEST PUT http://localhost:9002/.settings/rules.json?ns=emulator-test-1

[2020-03-02T11:10:05.992Z] <<< HTTP RESPONSE 400 content-length=38, content-type=application/json; charset=utf-8, access-control-allow-origin=*, cache-control=no-cache
, x-firebase-project-id=emulator-test-1, x-firebase-project-number=123456789, x-firebase-uuid=7bbc1cd3-3611-4a4a-b866-ea7f56b3ff1a
[2020-03-02T11:10:05.992Z] <<< HTTP RESPONSE BODY {
  "error" : "No data supplied."
}

!  database.rules.json:No data supplied.
!  Failed to update rules
i  database: Change detected, updating rules...
[2020-03-02T11:10:07.895Z] >>> HTTP REQUEST PUT http://localhost:9002/.settings/rules.json?ns=emulator-test-1
 {
  "rules": {
    ".read": true,
    ".write": true,
    "dinosaurs": {
      ".indexOn": [
        "height",
        "length"
      ]
    }
  }
}
[2020-03-02T11:10:07.906Z] <<< HTTP RESPONSE 200 content-length=15, content-type=application/json; charset=utf-8, access-control-allow-origin=*, cache-control=no-cache
, x-firebase-project-id=emulator-test-1, x-firebase-project-number=123456789, x-firebase-uuid=7bbc1cd3-3611-4a4a-b866-ea7f56b3ff1a
+  database: Rules updated.
i  database: Change detected, updating rules...
[2020-03-02T11:10:09.498Z] >>> HTTP REQUEST PUT http://localhost:9002/.settings/rules.json?ns=emulator-test-1

[2020-03-02T11:10:09.504Z] <<< HTTP RESPONSE 400 content-length=38, content-type=application/json; charset=utf-8, access-control-allow-origin=*, cache-control=no-cache
, x-firebase-project-id=emulator-test-1, x-firebase-project-number=123456789, x-firebase-uuid=7bbc1cd3-3611-4a4a-b866-ea7f56b3ff1a
[2020-03-02T11:10:09.504Z] <<< HTTP RESPONSE BODY {
  "error" : "No data supplied."
}

!  database.rules.json:No data supplied.
!  Failed to update rules

So it seems that sometimes it works, sometimes it doesn't.

@samtstern
Copy link
Contributor

@filipesilva thanks for being proactive here! This subset of logs shows that there is something flaky going on:

i  database: Change detected, updating rules...
[2020-03-02T11:09:57.032Z] >>> HTTP REQUEST PUT http://localhost:9002/.settings/rules.json?ns=emulator-test-1
 {
  "rules": {
    ".read": true,
    ".write": true,
    "dinosaurs": {
      ".indexOn": [
        "height",
        "length"
      ]
    }
  }
}
[2020-03-02T11:09:57.390Z] <<< HTTP RESPONSE 200 content-length=15, content-type=application/json; charset=utf-8, access-control-allow-origin=*, cache-control=no-cache
, x-firebase-project-id=emulator-test-1, x-firebase-project-number=123456789, x-firebase-uuid=7bbc1cd3-3611-4a4a-b866-ea7f56b3ff1a
+  database: Rules updated.
i  database: Change detected, updating rules...
[2020-03-02T11:09:59.724Z] >>> HTTP REQUEST PUT http://localhost:9002/.settings/rules.json?ns=emulator-test-1

[2020-03-02T11:09:59.733Z] <<< HTTP RESPONSE 400 content-length=38, content-type=application/json; charset=utf-8, access-control-allow-origin=*, cache-control=no-cache
, x-firebase-project-id=emulator-test-1, x-firebase-project-number=123456789, x-firebase-uuid=7bbc1cd3-3611-4a4a-b866-ea7f56b3ff1a
[2020-03-02T11:09:59.734Z] <<< HTTP RESPONSE BODY {
  "error" : "No data supplied."
}

It seems that sometimes when we try to read the file we're getting no content. Maybe that's a result of trying to read too soon after detecting a change?

@filipesilva
Copy link
Author

filipesilva commented Mar 3, 2020

In my case I saved the file but didn't change anything about it, so I'd expect an early read to return the same thing.

I did some more debugging locally by opening node_modules/firebase-tools/lib/emulator/databaseEmulator.js and modifying the source to this:

    start() {
        return __awaiter(this, void 0, void 0, function* () {
            if (this.args.rules && this.args.projectId) {
                const rulesPath = this.args.rules;
                this.rulesWatcher = chokidar.watch(rulesPath, { persistent: true, ignoreInitial: true });
                this.rulesWatcher.on("change", (event, stats) => __awaiter(this, void 0, void 0, function* () {
                    const newContent = fs.readFileSync(rulesPath).toString();
                    console.log('####')
                  console.log(fs.readFileSync(rulesPath))
                  console.log('####')
                  console.log(fs.readFileSync(rulesPath, 'utf-8'))
                  console.log('####')
                  console.log(newContent)
                    console.log('####')
                    utils.logLabeledBullet("database", "Change detected, updating rules...");
                    try {
                        yield this.updateRules(newContent);
                        utils.logLabeledSuccess("database", "Rules updated.");
                    }
                    catch (e) {
                        utils.logWarning(this.prettyPrintRulesError(rulesPath, e));
                        utils.logWarning("Failed to update rules");
                    }
                }));
            }
            return downloadableEmulators.start(types_1.Emulators.DATABASE, this.args);
        });
    }

Then, saving without changes again, I see the following output:

kamik@RED-X1C6 MINGW64 /d/sandbox/firebase-emulator-orderby-index (master)
$ yarn emulators
yarn run v1.21.1
$ yarn firebase emulators:start --project emulator-test-1
$ D:\sandbox\firebase-emulator-orderby-index\node_modules\.bin\firebase emulators:start --project emulator-test-1
i  emulators: Starting emulators: database, hosting
+  hub: emulator hub started at http://localhost:4400
i  database: database emulator logging to database-debug.log
+  database: database emulator started at http://localhost:9002
i  database: For testing set FIREBASE_DATABASE_EMULATOR_HOST=localhost:9002
i  hosting: Serving hosting files from: public
+  hosting: Local server: http://localhost:5002
+  hosting: hosting emulator started at http://localhost:5002
+  All emulators started, it is now safe to connect.
####
<Buffer 7b 0a 20 20 22 72 75 6c 65 73 22 3a 20 7b 0a 20 20 20 20 22 2e 72 65 61 64 22 3a 20 74 72 75 65 2c 0a 20 20 20 20 22 2e 77 72 69 74 65 22 3a 20 74 72 ... 97 mor
e bytes>
####
{
  "rules": {
    ".read": true,
    ".write": true,
    "dinosaurs": {
      ".indexOn": [
        "height",
        "length"
      ]
    }
  }
}
####
{
  "rules": {
    ".read": true,
    ".write": true,
    "dinosaurs": {
      ".indexOn": [
        "height",
        "length"
      ]
    }
  }
}
####
i  database: Change detected, updating rules...
+  database: Rules updated.
####
<Buffer 7b 0a 20 20 22 72 75 6c 65 73 22 3a 20 7b 0a 20 20 20 20 22 2e 72 65 61 64 22 3a 20 74 72 75 65 2c 0a 20 20 20 20 22 2e 77 72 69 74 65 22 3a 20 74 72 ... 97 mor
e bytes>
####
{
  "rules": {
    ".read": true,
    ".write": true,
    "dinosaurs": {
      ".indexOn": [
        "height",
        "length"
      ]
    }
  }
}
####

####
i  database: Change detected, updating rules...
!  database.rules.json:No data supplied.
!  Failed to update rules
####
<Buffer 7b 0a 20 20 22 72 75 6c 65 73 22 3a 20 7b 0a 20 20 20 20 22 2e 72 65 61 64 22 3a 20 74 72 75 65 2c 0a 20 20 20 20 22 2e 77 72 69 74 65 22 3a 20 74 72 ... 97 mor
e bytes>
####
{
  "rules": {
    ".read": true,
    ".write": true,
    "dinosaurs": {
      ".indexOn": [
        "height",
        "length"
      ]
    }
  }
}
####

####
i  database: Change detected, updating rules...
!  database.rules.json:No data supplied.
!  Failed to update rules
####
<Buffer 7b 0a 20 20 22 72 75 6c 65 73 22 3a 20 7b 0a 20 20 20 20 22 2e 72 65 61 64 22 3a 20 74 72 75 65 2c 0a 20 20 20 20 22 2e 77 72 69 74 65 22 3a 20 74 72 ... 97 mor
e bytes>
####
{
  "rules": {
    ".read": true,
    ".write": true,
    "dinosaurs": {
      ".indexOn": [
        "height",
        "length"
      ]
    }
  }
}
####

####
i  database: Change detected, updating rules...
!  database.rules.json:No data supplied.
!  Failed to update rules
####
<Buffer 7b 0a 20 20 22 72 75 6c 65 73 22 3a 20 7b 0a 20 20 20 20 22 2e 72 65 61 64 22 3a 20 74 72 75 65 2c 0a 20 20 20 20 22 2e 77 72 69 74 65 22 3a 20 74 72 ... 97 mor
e bytes>
####
{
  "rules": {
    ".read": true,
    ".write": true,
    "dinosaurs": {
      ".indexOn": [
        "height",
        "length"
      ]
    }
  }
}
####

####
i  database: Change detected, updating rules...
!  database.rules.json:No data supplied.
!  Failed to update rules

So it seems reading the buffer and converting it to a string is a problem. Not super sure why though.

@filipesilva
Copy link
Author

filipesilva commented Mar 3, 2020

I'm reluctant to say that using fs.readFileSync(rulesPath, 'utf-8') is the correct approach here, because I don't understand why .toString() on what ostensibly looks like the same buffer would return an empty string.

@Elyx0
Copy link

Elyx0 commented Mar 3, 2020

I have the same issue in the emulator.
I even tried to force the rules to be loaded with

    await firebaseTesting.loadDatabaseRules({
      databaseName,
      rules: fs.readFileSync('../database.rules.json', 'utf8');
    });

It still tells me in the emulator that my index is not there. @firebase/database: FIREBASE WARNING: Using an unspecified index.

   "@firebase/testing": "^0.16.13",
    "firebase": "^7.9.3",
    "firebase-admin": "^8.9.2",
    "firebase-functions": "^3.3.0",
    "firebase-functions-test": "^0.2.0",
    "firebase-tools": "^7.14.0",

firebase --version
7.14.0

@samtstern
Copy link
Contributor

@filipesilva hmmm that's really strange but I do appreciate the investigation and it sounds like we have a possible fix. @abeisgoat do you understand this?

@yuchenshi
Copy link
Member

I took a look at the docs and it seems buf.toString() defaults to utf8 as well. However, I believe readFileSync(file, encoding) will decode the stream instead of a buffer, which may be the difference here. I suspect that we're probably affected some Windows-specific Node.js bugs on buffers, so I'd say we just go with the readFileSync(file, encoding) approach before we dig too deep into the rabbit hole.

@filipesilva
Copy link
Author

filipesilva commented Mar 17, 2020

Just tried testing on node and I don't see it at all:

const { readFileSync } = require('fs');
const filePath = './file.txt';

async function timeout(ms) {
  await new Promise(resolve => setTimeout(() => resolve(), ms));
}

async function test() {
  for (let index = 0; index < 10; index++) {
    console.log("loop start")
    console.log("readFileSync(filePath):", readFileSync(filePath));
    console.log("readFileSync(filePath).toString():", readFileSync(filePath).toString());
    console.log("readFileSync(filePath, 'utf-8')", readFileSync(filePath, 'utf-8'));
    await timeout(1000)
  }
}

test();
$ node test.js 
loop start
readFileSync(filePath): <Buffer 66 69 6c 65 2e 74 78 74 20 63 6f 6e 74 65 6e 74 73>
readFileSync(filePath).toString(): file.txt contents
readFileSync(filePath, 'utf-8') file.txt contents
loop start
readFileSync(filePath): <Buffer 66 69 6c 65 2e 74 78 74 20 63 6f 6e 74 65 6e 74 73>
readFileSync(filePath).toString(): file.txt contents
readFileSync(filePath, 'utf-8') file.txt contents
loop start
readFileSync(filePath): <Buffer 66 69 6c 65 2e 74 78 74 20 63 6f 6e 74 65 6e 74 73>
readFileSync(filePath).toString(): file.txt contents
readFileSync(filePath, 'utf-8') file.txt contents
loop start
readFileSync(filePath): <Buffer 66 69 6c 65 2e 74 78 74 20 63 6f 6e 74 65 6e 74 73>
readFileSync(filePath).toString(): file.txt contents
readFileSync(filePath, 'utf-8') file.txt contents
loop start
readFileSync(filePath): <Buffer 66 69 6c 65 2e 74 78 74 20 63 6f 6e 74 65 6e 74 73>
readFileSync(filePath).toString(): file.txt contents
readFileSync(filePath, 'utf-8') file.txt contents
loop start
readFileSync(filePath): <Buffer 66 69 6c 65 2e 74 78 74 20 63 6f 6e 74 65 6e 74 73>
readFileSync(filePath).toString(): file.txt contents
readFileSync(filePath, 'utf-8') file.txt contents
loop start
readFileSync(filePath): <Buffer 66 69 6c 65 2e 74 78 74 20 63 6f 6e 74 65 6e 74 73>
readFileSync(filePath).toString(): file.txt contents
readFileSync(filePath, 'utf-8') file.txt contents
loop start
readFileSync(filePath): <Buffer 66 69 6c 65 2e 74 78 74 20 63 6f 6e 74 65 6e 74 73>
readFileSync(filePath).toString(): file.txt contents
readFileSync(filePath, 'utf-8') file.txt contents
loop start
readFileSync(filePath): <Buffer 66 69 6c 65 2e 74 78 74 20 63 6f 6e 74 65 6e 74 73>
readFileSync(filePath).toString(): file.txt contents
readFileSync(filePath, 'utf-8') file.txt contents
loop start
readFileSync(filePath): <Buffer 66 69 6c 65 2e 74 78 74 20 63 6f 6e 74 65 6e 74 73>
readFileSync(filePath).toString(): file.txt contents
readFileSync(filePath, 'utf-8') file.txt contents

This really surprises me though. I wonder if some dependency that firebase-tools uses is wrapping the native FS calls or something like that...

@samtstern
Copy link
Contributor

I am going to send a PR to add explicit encoding to all readFileSync calls in this codebase.

Also I think it should be utf8 not utf-8 but I guess Node is forgiving?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants