Firestore emulator sometimes fails on data export when stopped

[madmacc]

Describe your environment:

• Operating System version: Windows 10
• Browser version: Chrome 88.0.4324.104
• Firebase SDK version: 8.0.0
• Firebase Product: Firestore local emulator

Problem:
When I stop my local firestore emulator most of the time it exports the data fine. However sometimes is has an error and it deletes my export files so I lose all my data. Nothing seems to appear in the logs.
I am using:
firebase emulators:start --import=emulator/emulator-export-data --export-on-exit=emulator/emulator-export-data

Then:
Ctrl+C
to stop the emulator.

I get the following in the terminal:

i emulators: Received SIGINT (Ctrl-C) for the first time. Starting a clean shutdown.
i emulators: Please wait for a clean shutdown or send the SIGINT (Ctrl-C) signal again to stop right now.
i Automatically exporting data using --export-on-exit "emulator/emulator-export-data" please wait for the export to finish...
i Found running emulator hub for project hivepass-dev at http://localhost:4401
i Deleting directory R:\hivepass\HivePassApp\emulator\emulator-export-data\firestore_export
i Exporting data to: R:\hivepass\HivePassApp\emulator\emulator-export-data
! Export request failed, see emulator logs for more information.
! Automatic export to "emulator/emulator-export-data" failed, going to exit now...
i emulators: Shutting down emulators.
i ui: Stopping Emulator UI
! Emulator UI has exited upon receiving signal: SIGINT
i hosting: Stopping Hosting Emulator
i firestore: Stopping Firestore Emulator
! Firestore Emulator has exited upon receiving signal: SIGINT
i auth: Stopping Authentication Emulator
i hub: Stopping emulator hub
i logging: Stopping Logging Emulator

There is not much in the firestore-debug.log in the root:

INFO: Detected non-HTTP/2 connection.
Jan 28, 2021 12:46:37 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected non-HTTP/2 connection.
Jan 28, 2021 12:46:38 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected non-HTTP/2 connection.
Jan 28, 2021 12:46:38 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected non-HTTP/2 connection.
Jan 28, 2021 12:46:39 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected non-HTTP/2 connection.
Jan 28, 2021 12:46:39 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected non-HTTP/2 connection.

[schmidt-sebastian]

@yuchenshi - Can you take a look at this?

[sam-gc]

Hi @madmacc, thanks for the report.

Would you mind expanding a bit about the steps you're taking before seeing this issue? For example, how much data is being saved? What SDKs are you using? Are you using any of the other emulators in conjunction with Firestore?

[madmacc]

Sure @samhorlbeck.
The data is only 130k.
I am running the following emulators. Firestore, Auth, Functions
However I am running the functions emulator as a separate instance as the functions folder is in a different project/location and I can't figure out how to run all of them with one command.

I am using v9.2.1 of the firebase-tools.

Let me know if you need any more information.

[sam-gc]

What is your general use case? Can you discern any particular action that reliably causes the data to not be saved correctly? Basically I'm just trying to get a sense of how we could start to debug this issue and potentially reproduce it.

[madmacc]

Unfortunately not. I'm sure there is some reason but it seems random to me at this stage.
I can run it loading the previously exported data. Add some more data and then stop it by hitting ctrl+c in the terminal.
It will sometimes export ok and sometimes not. It only deletes the firestore_export folder not the firebase-export-metadata.json file.

At the moment when it is successful I manually copy the exported data to a safe location so it is not deleted.

I was hoping the logs would show something but they don't unless I am looking in the wrong place. Root of the project firestore-debug.log.

I am just testing an web application locally using the firebase js sdk and angularfire.

[madmacc]

I just tried starting and stopping it about 10 times in a row and it worked every time.
I tried adding some data, adding data from my app and even putting the computer into sleep mode and coming back out.

So not sure what it is but it is either time related or caused by something else.

[madmacc]

I just had a similar issue after leaving my computer for a couple of hours in sleep mode.
Slightly different error this time and at least it didn't delete the files this time!

Log is attached. Looks the same. I'm not sure if this is correct log file as it doesn't have much interesting in it.

i emulators: Received SIGINT (Ctrl-C) for the first time. Starting a clean shutdown.
i emulators: Please wait for a clean shutdown or send the SIGINT (Ctrl-C) signal again to stop right now.
i Automatically exporting data using --export-on-exit "emulator/emulator-export-data" please wait for the export to finish...
! Did not find any running emulators for project hivepass-dev.
! Automatic export to "emulator/emulator-export-data" failed, going to exit now...
i emulators: Shutting down emulators.
i ui: Stopping Emulator UI
! Emulator UI has exited upon receiving signal: SIGINT
i hosting: Stopping Hosting Emulator
i firestore: Stopping Firestore Emulator
! Firestore Emulator has exited upon receiving signal: SIGINT
i auth: Stopping Authentication Emulator
i hub: Stopping emulator hub
! hub: Error stopping emulator hub
i logging: Stopping Logging Emulator
firestore-debug.log

[yuchenshi]

! Did not find any running emulators for project hivepass-dev.

The Emulator Suite writes a temporary file to keep track of the running instances, and looks like that file was deleted somehow before you terminated the suite. For example, Windows may delete those temporary files to reclaim disk space (especially if disk space is low), or it may be some other cleanup tools installed on your dev machine.

Right now, --export-on-exit relies on that file, but it doesn't have to -- I've created b/179048252 to track making it more reliable. Thanks again for the bug report!

[madmacc]

@yuchenshi Ok that 2nd error above is because I am running two instances of the emulator. If I stop the functions emulator first it causes that problem. This is not the problem where it deletes the exported data files, it just fails to export.
My cloud functions folder is in a different project directory so I have to run the functions emulator as a separate instance.

Unfortunately it doesn't resolved my original problem of it deleting the files though.

[yuchenshi]

Glad we found that out! The export APIs cannot handle two instances running together yet, but the internal bug I've created above should be able to solve this.

In the meantime, have you considered creating symlinks so that your functions can be "part of" the main project directory? They are available on Windows 10 and you can ignore them in .gitignore (or similar) if you'd like to.

[madmacc]

In the meantime, have you considered creating symlinks so that your functions can be "part of" the main project directory? They are available on Windows 10 and you can ignore them in .gitignore (or similar) if you'd like to.

This is a good idea. I have not used them before but have had them on my todo list, I will give it a try. I tried a shortcut which of course did not work!

It must be common scenario for people that have multiple apps in their firebase project. Not sure why I am the only one who has brought it up.

[madmacc]

Actually after some more testing the first issue with the deleted export data files is due to the two instances of the emulator running.
But only if I start the firestore emulator first then the functions emulator and then stop the firestore emulator while the functions emulator is still running.

UPDATE:
It seems to work well with symbolic links. I will update if any issues.

If anyone else want to do this the command on Windows is:
mklink /D "R:\firestore-project\functions" "R:\cloud-functions-proj\functions"

[JoeEarly]

Am seeing this also on Mac, every other time my export files are either corrupted or missing data.

Will comeback with the detail

[carloschida]

On macOS 11.1 with firebase-tools@9.3.0 I'm seeing something of the kind:

On WebStorm 2020.3.2 when using the "Run" button on emulators:start:

// package.json

    "emulators:start": "firebase emulators:start --project=\"local-emulators\" --import=\"./a\" --export-on-exit=\"./a\""

The result is:

/Users/carlos.chida/.nvm/versions/node/v12.13.0/bin/node /usr/local/Cellar/yarn/1.22.5/libexec/bin/yarn.js run emulators:start
yarn run v1.22.10
$ firebase emulators:start --project="local-emulators" --import="./a" --export-on-exit="./a"
...
┌─────────────────────────────────────────────────────────────┐
│ ✔  All emulators ready! It is now safe to connect your app. │
│ i  View Emulator UI at http://localhost:4000                │
└─────────────────────────────────────────────────────────────┘

### STOPPED THE PROCESS VIA WEBSTORM ###

i  emulators: Received SIGINT (Ctrl-C) for the first time. Starting a clean shutdown.
i  emulators: Please wait for a clean shutdown or send the SIGINT (Ctrl-C) signal again to stop right now.
i  Automatically exporting data using --export-on-exit "./a" please wait for the export to finish...
⚠  Emulator UI has exited upon receiving signal: SIGINT
i  Found running emulator hub for project local-emulators at http://localhost:4400
i  Creating export directory /Users/carlos.chida/WebstormProjects/veriphi/functions/a
i  Exporting data to: /Users/carlos.chida/WebstormProjects/veriphi/functions/a

But only the folder gets created without any contents. I'm assuming that WebStorm is not giving the process enough time after sending the interrupt signal.

On the other hand, when running yarn emulators:start, it works without problems. Nonetheless, it looks like it consumes much time after receiving the signal:

yarn emulators:start
yarn run v1.22.10
$ firebase emulators:start --project="local-emulators" --import="./a" --export-on-exit="./a"
...
### PRESSED ctrl-C 
i  emulators: Received SIGINT (Ctrl-C) for the first time. Starting a clean shutdown.
i  emulators: Please wait for a clean shutdown or send the SIGINT (Ctrl-C) signal again to stop right now.
i  Automatically exporting data using --export-on-exit "./import-export-data" please wait for the export to finish...
i  Found running emulator hub for project local-emulators at http://localhost:4400                                                                                                                                     
i  Creating export directory /Users/carlos.chida/WebstormProjects/veriphi/functions/import-export-data
i  Exporting data to: /Users/carlos.chida/WebstormProjects/veriphi/functions/import-export-data
i  emulators: Received export request. Exporting data to /Users/carlos.chida/WebstormProjects/veriphi/functions/import-export-data.
$ ### NEW PROMPT!
➜ ✔  emulators: Export complete.
✔  Export complete
i  emulators: Shutting down emulators.
i  ui: Stopping Emulator UI
⚠  Emulator UI has exited upon receiving signal: SIGINT
i  functions: Stopping Functions Emulator
i  database: Stopping Database Emulator
i  firestore: Stopping Firestore Emulator
i  pubsub: Stopping Pub/Sub Emulator
⚠  functions: The Cloud Firestore emulator is not running, so calls to Firestore will affect production.
i  auth: Stopping Authentication Emulator
i  hub: Stopping emulator hub
$ ### ANOTHER NEW PROMPT

[yuchenshi]

@carloschida Thanks for the report! Exporting may take a few seconds and if WebStorm isn't giving the emulator enough time to do the work, there isn't a ton we can do here. We suggest running emulators:start in a terminal and you may find the built-in WebStorm terminal emulator better than the "Run" button for your case. Anyway, feel free to contact WebStorm regarding the issue.

[MorenoMdz]

please fix exporting I have this error plus another error in another issue. Clearly export does not work correctly for a lot of people and needs more testing.

Indeed. We plan on fixing this in our project by moving of Firestore.

[arisAlexis]

I opened a new issue here with multiple bugs also including this one. #5404

[bmcbarron]

Based on my experimentation, it looks like Vite dev server holds a lock on folders that contain sub-folders (at least on Windows). Filed bug report vitejs/vite#11938

[sugoidesune]

So I run "firebase emulators:start --import firestore_temp --export-on-exit"

1. It deletes firestore_temp
2. It creates it's own folders
3. It keeps creating new ones, never importing data.
   

I only have the functions and emulator running in there...

[ahmednfwela]

@sugoidesune a temporary workaround is to change save path to somewhere on c:/ drive, e.g. c:/firebase/exports

[sugoidesune]

@sugoidesune a temporary workaround is to change save path to somewhere on c:/ drive, e.g. c:/firebase/exports

Did you manage to do relative paths. ../folder didn't work for me either.

[TheoOliveira]

@sugoidesune a temporary workaround is to change save path to somewhere on c:/ drive, e.g. c:/firebase/exports

in linux would that work?

[bmcbarron]

If, like me, your symptoms are caused by the Vite dev server holding locks on nested folders (which can prevent the emulator from renaming its snapshots on Windows), the workaround I'm using is to set server: { watch: { usePolling: true }} in vite.config.ts. That changes the implementation of the directory watching and I haven't had issues ever since.

[twilliams0001]

If, like me, your symptoms are caused by the Vite dev server holding locks on nested folders (which can prevent the emulator from renaming its snapshots on Windows), the workaround I'm using is to set server: { watch: { usePolling: true }} in vite.config.ts. That changes the implementation of the directory watching and I haven't had issues ever since.

This worked for me. Thanks!

[YKakdas]

Creating a VHD and using it as the export folder worked for me. No matter what I tried, nothing worked when I only had 1 disk drive partition.

[joe528]

The export always succeeds if I launch the emulator a few minutes/hours ago.

But if I leave the emulator running for an extensive time (maybe 1-2 days) on my Mac, the export fails always.

Hope this can get fixed. I just lost data again today since I forgot about this issue.

[dennismuench]

It is happening no matter what (MacOS). For me this feature never worked. I have this exact problem since I first started experimenting with Firebase years ago. I had this problem on Windows, Mac and with every framework. Even if I just run a static SPA through the hosting emulator (no active dev server). Compiling the comments so far it looks like it is firebase-tools own file change detection mechanism that is locking things up...

[SKempin]

Running firebase emulators:export ./tmp results in the following log error for me. The dir is created but is empty. Any further progress with this?

⚠ emulators: Export failed: ENOENT: no such file or directory, mkdtemp 'firebase-export-1691618521143XXXXXX'

[christhompsongoogle]

SKempin, the emulators create the export in a temp directory and then copy them over once completed:

firebase-tools/src/emulator/hubExport.ts

Line 134 in 40dd38f

fse.moveSync(this.tmpDir, this.exportPath);

The mkdtemp creation is failing for some reason, inside of /tmp

See: https://www.geeksforgeeks.org/node-js-fs-mkdtempsync-method/

Are you running this inside a container or with permissions that would prevent the firebase CLI from creating a temp directory in /tmp?

[SKempin]

@christhompsongoogle thanks, no not running in a container and have also tried running with sudo, and multiple different directory paths. Same result each time.

[FAR5HID]

So I run "firebase emulators:start --import firestore_temp --export-on-exit"

1. It deletes firestore_temp
2. It creates it's own folders
3. It keeps creating new ones, never importing data.
   

I only have the functions and emulator running in there...

Same situation when running in container

[HirukaChansilu]

If, like me, your symptoms are caused by the Vite dev server holding locks on nested folders (which can prevent the emulator from renaming its snapshots on Windows), the workaround I'm using is to set server: { watch: { usePolling: true }} in vite.config.ts. That changes the implementation of the directory watching and I haven't had issues ever since.

Thanks, This worked also for me.

[remillc]

No comments since october 2023, am I missing something?

This problem is major for the developer experience.

For me, changing the path to C:\firebase-temp-data did the trick.

[yanir21]

No comments since october 2023, am I missing something?

This problem is major for the developer experience.

For me, changing the path to C:\firebase-temp-data did the trick.

Still not working for me. Does anyone have a different workaround?

[SKempin]

@yanir21 I am running this npm script now to run the emulator and it is successfully exporting on every stop, as long as I stop using ctrl + c

"serve": "npm run build && firebase emulators:start --import emulator_data --export-on-exit emulator_data"

I have a emulator_data dir at root level of the project.

[yanir21]

@SKempin Still no luck:

...
i  Exporting data to: C:\Users\USER\Desktop\functions\emulator_data
i  emulators: Received export request. Exporting data to C:\Users\USER\Desktop\functions\emulator_data.
!  emulators: Export failed: EPERM: operation not permitted, rename 'firebase-export-1707209884206CzcLW7' -> 'C:\Users\USER\Desktop\functions\emulator_data'
!  Export request failed, see emulator logs for more information.
!  Automatic export to "emulator_data" failed, going to exit now...
i  emulators: Shutting down emulators.
...

[SKempin]

@yanir21 can you push a repo for me to try this?

[mognutecnologia]

Hello, in my case it fails exactly as described by Mainul Islam Farshid, but I found out that my "running server" process was holding the lock of the whole directory. In someone's case there could be even others holding the lock. So what I have been doing until firebase team solves this: 1 - Before starting the emulator, I check if last export succeeded, if not, I manually rename the failed exported directory to "firebase-export-data" 2 - Start emulator making sure the export name matches the import argument when starting emulator: eg: " --import firebase-export-data"

firebase emulators:start --import ./firebase-export-data --export-on-exit ./firebase-export-data --inspect-functions

3 - Work 4 - Before exiting, I stop serving, compiling, and everything else that may be holding the lock *before *I exit the emulator. 5 - Making sure the emulator is the last running process, I kill it with *one single* ctrl+c, and wait for it to export (can take a few seconds, so don't hit several ctrl+C). 6 - I check the directory, if I still have my "firebase-export-data" folder, if it fails it will change to something like: "firebase-export-1707209884206CzcLW7". In this case I manually rename it to "firebase-export-data", to make sure I don't forget next time I will start the emulator (step 1). Needs a bit of extra attention, but works for me.

…

On Tue, Feb 6, 2024 at 7:34 AM Stephen Kempin ***@***.***> wrote: @yanir21 <https://github.com/yanir21> can you push a repo for me to try this? — Reply to this email directly, view it on GitHub <#3092 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQKYLEXTLMCUJKNUGMVKO73YSIBMHAVCNFSM4WX2OA6KU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJSHEZDEMZXG43A> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[yanir21]

I finally managed to solve this 🎉

The root directory (e.g /project ) holds the firebase.json file, but I tried to run the emulators from /project/functions because that's where my package.json npm scripts are.

Running emulators:start from the root project directory successfully exports the data to the specified folder, and also shuts down gracefully (which is another problem I had).

So changing the serve script to the following:

"serve": "npm run build && cd .. && firebase emulators:start --import ./emulator-data --export-on-exit ./emulator-data --inspect-functions"

solved it for me.

[lssilveira11]

@yanir21 solution worked for me too, thanks!

[reimertz]

Our issue was relying on concurrently to spawn multiple processes (used to build typescript).

The issue is twofold;

• When sigterming concurrently it sends multiple sigterm downwards to processes it has initiated.
• If firebase-tools recieve more than one CTRL-C, it force-exits without shutting down gracefully.

This has the side-effect of import / export not gracefully exporting and explains why different solutions such as disabling storage, running the command in different folders etc. might result in the import / export being able to finish before being force-quit.

I hope this can help you all!

[ngblaylock]

Our issue was relying on concurrently to spawn multiple processes (used to build typescript).

The issue is twofold;

• When sigterming concurrently it sends multiple sigterm downwards to processes it has initiated.
• If firebase-tools recieve more than one CTRL-C, it force-exits without shutting down gracefully.

This has the side-effect of import / export not gracefully exporting and explains why different solutions such as disabling storage, running the command in different folders etc. might result in the import / export being able to finish before being force-quit.

I hope this can help you all!

This seems to answer my issue. I changed my npm script to be this:

Now all my logs can still be in the same terminal, and using command + c closes all of them and successfully completes the export. The logs are not quite as pretty, but my export works now which is more important.

[mbpictures]

Still having an error during export:

i  Creating export directory B:\Projects\DigitalChoirLeader\packages\firebase\firebase-export
i  Exporting data to: B:\Projects\DigitalChoirLeader\packages\firebase\firebase-export
i  emulators: Received export request. Exporting data to firebase\firebase-export.
!  emulators: Export failed: EPERM: operation not permitted, rename 'firebase\firebase-export-1709550643283fOGVYX' -> 'firebase\firebase-export'
!  Export request failed, see emulator logs for more information.
!  Automatic export to "./firebase-export" failed, going to exit now...
i  emulators: Shutting down emulators.
i  ui: Stopping Emulator UI
!  Emulator UI has exited upon receiving signal: SIGINT
i  firestore: Stopping Firestore Emulator
!  Firestore Emulator has exited upon receiving signal: SIGINT

Even trying to manually rename the specified folder failed with the error, that the contents of the directory are still in use by some program. This error happens on Windows as well as on WSL.

[JFGHT]

Our issue was relying on concurrently to spawn multiple processes (used to build typescript).

The issue is twofold;

• When sigterming concurrently it sends multiple sigterm downwards to processes it has initiated.
• If firebase-tools recieve more than one CTRL-C, it force-exits without shutting down gracefully.

This has the side-effect of import / export not gracefully exporting and explains why different solutions such as disabling storage, running the command in different folders etc. might result in the import / export being able to finish before being force-quit.

I hope this can help you all!

This worked.