Importing data via emulator UI causes massive memory usage

[nikmartin]

[REQUIRED] Environment info

firebase-tools: 8.4.0

Platform: Arch Linux

[REQUIRED] Test case

Export a modest dataset from an existing project. Im my case RTDB json export is 7.5 MB, and my functions count is about 95, with approximately 75% of those are triggers of some sort.

[REQUIRED] Steps to reproduce

Initialize the emulator for RTDB and functions, then start emulator. Import a 7.5mb json file into RTDB. Memory starts increasing at a rate of about 1gb per second, until my OOM manager starts killing processes. My project has about 95 functions, and 75% of those are database triggers. My assumption is that a lot of triggers are firing simultaneously.

[REQUIRED] Expected behavior

This may be a bug, but a feature would be a bootstrap mode that lets me import data without any database triggers firing.

[REQUIRED] Actual behavior

The emulator itself continues to run, by my OOM manager starts killing processes to try to accomodate the memory usage, until pretty much all processes have been killed. The app itself never throws any errors.

[google-oss-bot]

This issue does not have all the information required by the template. Looks like you forgot to fill out some sections. Please update the issue with more information.

[nikmartin]

Also, I do have debug logs, but would rather not post them here, but I'd be glad to share them in some slightly more secure method.

[samtstern]

@nikmartin thanks for filing this! I believe you also posted this on Twitter, so I recognize the issue.

There are two things we can do here:

1. Introduce a way to import data in the RTDB emulator without triggering Cloud Functions. That work is already underway.
2. Make sure the Functions emulator rate-limits itself better. I'm fine with some kind of failure in this case like a functions timeout but using all your memory is just really bad. I'll take this on.

[nikmartin]

I believe you also posted this on Twitter, so I recognize the issue.

Yes, I was having trouble finding where to report it, as Firebase has a few channels at the moment! I'll delete the Twitter comments as Twitter is really for marketing, not support

[samtstern]

@nikmartin no need to delete the Twitter comments! We're always happy to get good feedback on any channel. Although you're right that we don't actively monitor every reply to @Firebase.

[nikmartin]

In using the functions emulator a bit more, although it was cumbersome since i have ~100 firebase functions, I discovered process.env.FUNCTIONS_EMULATOR that I added a check for at the top of any functions that shouldn't run in the emulator. It's not the same as a bootstrap feature, but at least I'm not spamming all the devs with emails and push notifications in our app. In doing that, I also noticed the memory consumption isn't nearly as high on import .

[samtstern]

@nikmartin glad that you have a temporary workaround!

[nikmartin]

I think you probably have a handle on this, but just to add more info, if I run the emulator with --inspect-functions, forcing functions to run serially, memory usage remains nominal while all the function database triggers fire

[samtstern]

@nikmartin I am working on a solution over here:
#2321

Let me know what you think.

[nikmartin]

That looks great, I think that will work perfectly.

[fpeterdev]

@samtstern
I was about to create a feature request for the option to disable RTDB triggers in the emulator during imports. While searching for possible duplicates I found your comment:
Introduce a way to

import data in the RTDB emulator without triggering Cloud Functions. That work is already underway.

Sooo... Is there any update on that feature? :) It would extremely useful!

[samtstern]

@fpeterdev yes it's not documented yet (on my to-do list!) but there is a function in @firebase/rules-unit-testing called withFunctionTriggersDisabled() that lets you do this.

Example:

const testing = require('@firebase/rules-unit-testing');

await testing.withFunctionTriggersDisabled(async () => {
  await doSomeDataThing();
  // ...
});

[fpeterdev]

Thank you! Will such an option be available from the UI during import? We are using the emulator for development as well, and it would be great if we could reload the initial data to the database without restarting the emulator.
By the way, this feature would make sense on the "real" firebase console as well. (e.g. for restoring a previous backup)