Driver: Firebase

[IagoLast]

I think that could be a nice addition since firebase is widely used and there's no dashboard or admin panel for it.

https://firebase.google.com/

⬇️ Please click the 👍 reaction instead of leaving a +1 or update? comment

[thomashs-zz]

Hey IagoLast. Would you join me on this?

[IagoLast]

@thomashs sure! But this is a Big thing. I don't know where to start and I think that kind of things require the aproval of a Core contributor right?

[thomashs-zz]

@IagoLast yeap. Here's how they suggest driver implementation: https://github.com/metabase/metabase/wiki/Writing-A-Driver

[IagoLast]

ummm I`m not sure about how this works... is some kind of meta-lang? @thomashs

[salsakran]

@IagoLast the doc @thomashs references describes how to write a driver.

All GUI queries are expressed in MBQL(https://github.com/metabase/metabase/wiki/Query-Language-'98) , and a "driver" is effectively a transpiler from MBQL to the target language.

I'll also note that any driver PR must have include integration tests against that data source, and a way for us to run them from CI.

[IagoLast]

@thomashs ok, lets try to do it. were do we start?

[SantiMunin]

I can't say for sure yet (I'd need approval from my company and some time that I might not have) but I'd like to join this effort.

At the very least I should be able to guide @IagoLast (I have some experience with sql-to-something compilation and we know each other, so it'd be a nice experience).

[prigal]

I have added my +1 yesterday, but after digging I'm not sure to really understand this issue.

Do you mean access to firebase database or firebase analytics ?

(I was interested in gathering analytics from firebase in metabase, and it looks like I can send my events to bigquery and then access bigquery from metabase to build my dashboard)

[thetwosents]

The database would be my vote. There are ways to get data from Firebase Analytics already but querying / visualizing your data isn't accessible from BI tools

[salsakran]

For what it's worth, for firebase specifically, we'd strongly suggest getting your database and/or analytics events into BigQuery somehow vs coding against the database rest api directly.

I'm leaving this issue open in case to not squash anyone's motivation, but even if it does get addressed in a PR, I'd still recommend using BQ =)

[IagoLast]

@pierrerigal I meant firebase database.

@salsakran I understand you mean to dump the data into a different database and then do BI from metabase against the "backup", I dind´t know that BigQuery was supported. I assume that BQ is automatically synchronized with firebase right?

I was doing something similar with mongoDB. I have my real data in firebase and a mongoDB copy where the BI work is done through metabase. This requires extra work to keep both databases synced. I think that it would be a nice adition being able to sync firebase directly.

[thetwosents]

It is not directly supported. I just looked :/

…

On Mar 14, 2017, at 11:56 AM, IagoLast ***@***.***> wrote: @pierrerigal <https://github.com/pierrerigal> I meant firebase database. @salsakran <https://github.com/salsakran> I understand you mean to dump the data into a different database and then do BI from metabase against the "backup", I dind´t know that BigQuery was supported. I assume that BQ is automatically synchronized with firebase right? I was doing something similar with mongoDB. I have my real data in firebase and a mongoDB copy where the BI work is done through metabase. This requires extra work to keep both databases synced. I think that it would be a nice adition being able to sync firebase directly. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4213 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADmHkQ6qtD5dj2U9sAJKTG-ngTEDbA4mks5rlsbSgaJpZM4Lqz4D>.

[IagoLast]

@thetwosents afaik BQ only supports synchronizing analytics data from firebase, but I want the actual database data from my app.

am I right?

[thetwosents]

Absolutely. The biggest business value for anyone in Firebase right now is to have a clear understanding of their data without having to build out an entire one off solution for it. Hands down.

…

On Mar 14, 2017, at 12:02 PM, IagoLast ***@***.***> wrote: @thetwosents <https://github.com/thetwosents> afaik BQ only supports synchronizing analytics data from firebase, but I want the actual database data from my app. am I right? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4213 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADmHkUNIkir27n8CySdsUkgW0-7ExMqSks5rlshAgaJpZM4Lqz4D>.

[thetwosents]

I’ve done exports and visualized in grow and charted in multiple libraries, but thankfully, metabase has a clear path for me and so I’d like to help however I can.

…

On Mar 14, 2017, at 12:02 PM, IagoLast ***@***.***> wrote: @thetwosents <https://github.com/thetwosents> afaik BQ only supports synchronizing analytics data from firebase, but I want the actual database data from my app. am I right? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4213 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADmHkUNIkir27n8CySdsUkgW0-7ExMqSks5rlshAgaJpZM4Lqz4D>.

[salsakran]

What we've found working with companies on eg Parse, is that you're better off moving to a dedicated data warehouse + ETL (or ELT) world sooner rather than later.

At a certain stage in the company lifecycle, the benefits of having all of your data in one place and in a semi-reconciled format outweighs the hassle of getting it all in there. This keep events and your application database in a place where you can cross reference them, do transforms or joins on them, etc. It simplifies analyst (and end user) workflows, and allows more sophisticated work faster.

With DaaS's (Database as a service with custom APIs), the hassle of writing a connector (or driver in our parlance) is usually a bit of a drag. Given that eventually most people needs to get data out of said DaaS and into a real data warehouse to do any real work, we've found people happier when they "graduate" to a real data warehouse sooner than folks on MySQL or PostgreSQL for example.

I'll also note that It would be much less work to write a small job that dumps firebase database data onto Google Cloud Storage, and cron's an ingestion than writing a MB driver.

[thetwosents]

I agree with that. I guess the truth is, the migration to a larger data house is definitely the way to go, but the benefit of firebase is ten fold when starting out. Its both sides I think. Do you happen to have an ingest script that would could make available for metabase / firebase users?

…

On Mar 14, 2017, at 12:23 PM, Sameer Al-Sakran ***@***.***> wrote: What we've found working with companies on eg Parse, is that you're better off moving to a dedicated data warehouse + ETL (or ELT) world sooner rather than later. At a certain stage in the company lifecycle, the benefits of having all of your data in one place and in a semi-reconciled format outweighs the hassle of getting it all in there. This keep events and your application database in a place where you can cross reference them, do transforms or joins on them, etc. It simplifies analyst (and end user) workflows, and allows more sophisticated work faster. With DaaS's (Database as a service with custom APIs), the hassle of writing a connector (or driver in our parlance) is usually a bit of a drag. Given that eventually most people needs to get data out of said DaaS and into a real data warehouse to do any real work, we've found people happier when they "graduate" to a real data warehouse sooner than folks on MySQL or PostgreSQL for example. I'll also note that It would be much less work to write a small job that dumps firebase database data onto Google Cloud Storage, and cron's an ingestion than writing a MB driver. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4213 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADmHkR7EiHIgYUXBPxjT4qLbyFYRfXtsks5rls0qgaJpZM4Lqz4D>.

[frapsMatheus]

This would be great, just exported my firebase to BigQuery but the only useful way to use the app events on it is trough SQL Querys.

[joshuarcher]

@matrpedreira How'd you get firestore exporting to BigQuery?

[kamikazechaser]

Firebase now has 2 database types: Firestore and Realtime database.

Not viable with Firestore. Firestore has a r/w pricing model. Even trying to sync a periodically dumped JSON will be expensive, not to mention the complex pipeline you have to build just to visualize the data.

With Real time database, cost is not an issue. But you still have to introduce the complexity of a pipeline to sync with BigQuery.

You can directly integrate analytics with BigQuery and it will sync. So only analytics is possible that through BigQuery.

I don't see this being viable for Metabase.

[frapsMatheus]

Hi @joshuarcher. This tutorial is going to help you.
https://support.google.com/firebase/answer/6318765?hl=pt

[oodavid]

In 2021 Firebase has a built-in BigQuery integration:

https://support.google.com/firebase/answer/6318765

Looks as simple as flipping a switch in the Firebase console:

I will try this today and report back

[oodavid]

Boo, it doesn't include Realtime DB or Firestore :(

[oodavid]

Turns out there's an extension:

https://firebase.google.com/products/extensions/firestore-bigquery-export

[jamesgreenblue]

Following the advice of @salsakran we are using BigQuery as an intermediary between Firestore (in datastore mode) and Metabase. We are hitting issues with primary and foreign keys, have written this up in the discourse:

https://discourse.metabase.com/t/firestore-big-query-metabase-issue-with-joining-and-displaying-foreign-key-name-value/24051

For the people commenting in this thread, have you managed to get a pipeline between Firestore and Metabase working? I hope I'm missing something obvious (we are using Firestore in Datastore mode).