What's the best way to migrate firestore db to supabase?

[inbytesdevteam]

Question

What's the best way to migrate firestore db to supabase?

Additional context

We are trying out supabase to become our backend. We have some data already stored in our firestore db but it is in NoSQL.

So our strategy is:

• Extract all our db into a JSON format.
• Convert the JSON data types to match supabase DB fields and data types.
• Upload the updated JSON db to supabase table using a script.

Is there a simpler way or a feature at supabase where we can import JSON files from firebase/firestore and it auto-converts to supabase db? Or this is the best strategy for now?

Thanks. :)

[awalias]

hey @inbytesdevteam

I think your current approach is most likely the best one. We are about to ship a CSV to SQL Table converter but I think if you're going to the effort of converting to CSV then you may as well just convert directly to Supabase.

We haven't attempted this migration path ourselves yet and will be keen to hear how you get on, I will leave this open to see if anyone else on the team has some useful input here.

[thorwebdev]

@inbytesdevteam is your data in Firebase Realtime DB or Cloud Firestore?

If Realtime DB, the following workflow might work:

• Export your Data as JSON (Go to Firebase console > Database > Realtime Database and click on Export JSON)
• Set up a hasura graphql engine and set your Supabase database as DATABASE_URL env var: https://github.com/hasura/graphql-engine#other-one-click-deployment-options
• Run https://github.com/hasura/firebase2graphql

For Cloud Firestore there currently isn't a way to export JSON, see this related issue here: hasura/firebase2graphql#3

This might be an interesting hacktoberfest project: create a Firebase Cloud Function that exports your Firebase Realtime DB or Cloud Firestore data to Supabase.

[inbytesdevteam]

Hi @thorwebdev we are currently using Cloud Firestore.

For Cloud Firestore, I am using firestore-export-import to export it to JSON format.

I just need to redesign the JSON output to have the same fields in our supabase db and convert the timestamp to be the same data type as well.

I do have a question about the data types allowed in Supabase, do we have an official list somewhere? Can we set an array as a data type? Can we add the official list of data types in the documentation as well so we can use it as basis when we are creating/updating our columns. Thank you!

PS: Nice idea for a hacktoberfest! :)

[kiwicopple]

2023 update:

you can use either the migration tool:

https://github.com/supabase-community/firebase-to-supabase

Or you can use the firebase wrapper:

https://supabase.github.io/wrappers/firebase/

There's also a neat tool to help with the migration called "supashim": https://github.com/roboflow/supashim

---

2020 response:

Hey @inbytesdevteam, I did this in my previous startup. It is pretty much the origin story of Supabase.

There are 2 considerations:

1. Migrating your data
2. Keeping both systems in sync (if you cannot make a clean switch over, and you have to run them in parallel)

1. Migrating your data

You've got the right idea. The steps are:

1. Export your data from Firebase. I used firestore-backup to dump everything into JSON files
2. In Supabase, create a table for each "root" collection that you have in firebase. For example, if you have a "users" collection, create a users table in Supabase
3. Add a JSONB column to each table in step 2
4. Load the data into Supabase - each row contains a document, inserted into the JSONB column. This could be helpful.
5. If you want, you can use generated columns or views to make the JSONB data easier to query

2. Keeping both systems in sync

This is a bit more advanced. If you cannot switch your system over, then you should create a Firebase Function which listens to all inserts/updates on your documents, and then inserts it into your Supabase Postgres database.

Likewise, using our Realtime server you can listen to inserts/updates on your Postgres database and insert them back into Firebase.

You will need a column/key on each to specify the origin of the initial insert (Firebase/Supabase) so that you don't end up in an infinite loop of inserting/updating.

This gets a bit complicated with updates, so it really needs some planing out for your system.

Data Types

question about the data types allowed in Supabase, do we have an official list somewhere?

Good question! It's just Postgres, so you have a lot of available types: https://www.postgresql.org/docs/current/datatype.html. You can even create custom composite types: https://www.postgresql.org/docs/9.6/rowtypes.html

Also, with JSONB, you have all the flexibility of a JSON object.

Next steps

At some point we would like to write a migration tool for this, but we're not ready yet. Let us know if there is anything else we can help with in the meantime :)

[kasvith]

Hi @kiwicopple

In here, it mentions about a middleware to update passwords in supabase, but it doesn't mention how to integrate it. can you shed some light

https://github.com/supabase-community/firebase-to-supabase/tree/main/auth

[inbytesdevteam]

Thanks @kiwicopple. Cool to know that this is the origin story of Supabase and thanks for the data types reference. Was looking into what types we can use.

Thanks @thorwebdev and @kiwicopple for the help. We were able to easily move our db from firestore to supabase :)

[thorwebdev]

@inbytesdevteam that's awesome to hear 🎉

Would you mind outlining the process you went through to make it happen? Did you end up writing a custom script? And if so, could you add it to the examples? This would be super useful for others :)

[inbytesdevteam]

Hi @thorwebdev, sorry for the late revert. I wasn't able to create a custom script since it would be easier for me to set it up by designing each script tailored to the JSON file from Firestore and our db at supabase.

Step 1: Exporting the Data

I have used Firestore-Export-Import to export the data out of firestore.

Step 2: Read the JSON file, parse the JSON data and map it to it's respective variable based on your db design at supabase.

1. Read the JSON file using fs.readFile()
2. Parse the JSON data using JSON.parse()
3. Map the parsed JSON to respective variables using for loops, forEach and check if there is a need to convert it's data type.
   Example: Date. The output that we got from the exported firestore is an object with two keys with values of string. Had to convert this to Date first before pushing to supabase.

"start_date": {
    "_seconds": 1575149400,
    "_nanoseconds": 0
}

Step 3: Store them in an array and insert the array of objects to supabase using .insert;

var supabaseData = []

let supabaseObject = {
}
supabaseData.push(supabaseObject);

try {
      await supabase.from('databaseName').insert(supabaseData);
    } catch(err) {
      console.log(err); // TypeError: failed to fetch
    }

I can create one example template that people can use if you like. Let me know if you want. :)

[kiwicopple]

This is great thanks @inbytesdevteam - we are going to leave this open as a task for Hacktoberfest. If you want to solve it then, that would be great (otherwise perhaps someone else can use your instructions to build a template).

[kiwicopple]

We're outside of Hacktoberfest so I've converted this to a discussion

[chrismclarke]

Since the time of first writing firebase have also released their own methods to export data from firestore (https://firebase.google.com/docs/firestore/manage-data/export-import), although this does not export to json but instead a file format to keep compatibility with google cloud datastore and bigquery. There is also a difference if choosing to export the entire database (which currently I think can only be re-imported into firestore), or providing an explicit list of collections which can then be imported into bigquery.

So I'm not sure if that's also you would consider providing support for or not (my guess is it would be quite a bit of extra work).

[chrismclarke]

Just as a quick follow-up to this, I tried exporting data from firestore using the above export methods that generates a number of metadata and output files which are encoded for use by google's protobuf compiler.

Trying to decode them is extremely difficult (see issue at https://stackoverflow.com/questions/57441242/decoding-managed-protocol-buffer-datastore-firestore-export-by-google-with-proto), and involves downloading and recompiling various protoc files from the googleapis library, reading the data files in specific chunks and handling conversion.

One repo does already exist that attempts to do this in python (leveraging google app-engine code for reading in data, so not easily transferrable to other languages I think): https://github.com/labbots/firestore-export-json, but my own testing has found it doesn't work consistently well.

On the plus side, the exported files can be imported directly into the newly released firestore emulator, so you can test full data migration locally without the cost of server reads.

[baptisteArno]

Hey guys,

I'm the founder of Typebot and I'm currently implementing analytics-based features and this is where Firebase isn't suitable at all (I need to increment and decrement things in cloud functions all the time and it could become out of phase with the existing data).

So I'm thinking about migrating to Supabase. But, I currently have a large amount of data so it would be quite a challenge.

I'm thinking, instead of migrating everything with a one-shot process, maybe I would implement an app state where Firebase and Supabase are co-existing so that newly created users and assets would be saved in Supabase and the old data would still work with Firebase.

The only question I have is how will I handle Authentication with existing users because they will eventually interact with old and new data so they would basically have an auth state for Firebase and Supabase?

I'm wondering if it will work... Is it insane? 😅

[thorwebdev]

@baptisteArno that's an interesting problem, I'll drop down a couple ideas, maybe it can help you point in the right direction:

1. Not necessarily insane, just maybe need to test how well it performs. For new users everything lives in Supabase, easy. For existing user, sign them up with Supabase auth using their email and for the password value use a secret that you securely store in Firebase so only the authenticated user can access it. When an existing user logs in, first log them in with Firebase auth so they can access their Supabase auth password in Firestore. Then use that password to log them in with Supabase in the background.
2. Another option would be to split systems by users. So new users interact with Supabase while existing users and all their data stay in Firebase. That way you can approach your migration on a per user basis. I guess this isn't ideal though, as the analytics features would not be available for existing users, but maybe you could migrate users based on if they are asking for access to these features?
3. Keep creating users in Firebase Auth but create all new data in Supabase and use a cloud function to proxy the data from Supabase. The function would evaluate whether the user is authenticated and then use Supabase admin credentials to access their data. It's also not ideal, as all data would run through your function, so that's a consideration, but it could make the new features available for everyone at the same time if that's the priority.

[baptisteArno]

Thank you for those insights. I'll keep you in touch! 👍

[matthewwong525]

I know this has already been answered, but here's a link to my blog post about how I migrated from firebase to supabase. If you're looking to migrate I think this blog post will be super helpful :)

https://fleetingnotes.app/posts/migrating-from-firebase-to-supabase/