Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't import Firebase SDK #112

Closed
rick-stevens opened this issue Feb 2, 2019 · 24 comments
Labels

Comments

@rick-stevens
Copy link

@rick-stevens rick-stevens commented Feb 2, 2019

Related

https://stackoverflow.com/questions/51522280/use-firebase-sdk-with-netlify-lambda-functions

Dependencies

{
  "firebase": "^5.8.2",
  "firebase-admin": "^7.0.0",
  "netlify-lambda": "^1.3.1",
}

The issue

I'd love to use Firebase (Firestore/Storage) in my Netlify function, though it won't even let me import any related SDK using netlify-lambda serve. I've tried both firebase and firebase-admin which respectively result in these delayed (5 - 10 seconds) errors:

Using firebase:

import firebase from 'firebase/app'
import 'firebase/firestore'

exports.handler = (event, context, callback) => callback(null, {
  statusCode: 200,
  body: 'OK'
})
Output:
netlify-lambda: Starting server
Lambda server is listening on 9000
Hash: e3817619e6c76f23fc78
Version: webpack 4.29.0
Time: 3210ms
Built at: 02/02/2019 11:55:21 PM
      Asset      Size  Chunks             Chunk Names
firebase.js  2.72 MiB       0  [emitted]  firebase
Entrypoint firebase = firebase.js
 [10] external "fs" 42 bytes {0} [built]
 [12] ../node_modules/firebase-admin/lib/utils/error.js 34.8 KiB {0} [built]
 [14] ../node_modules/firebase-admin/lib/utils/validator.js 7.25 KiB {0} [built]
 [56] ../node_modules/firebase-admin/lib/utils/deep-copy.js 2.6 KiB {0} [built]
[115] ../node_modules/firebase-admin/lib/auth/credential.js 15 KiB {0} [built]
[235] ../node_modules/firebase-admin/lib/index.js 1.36 KiB {0} [built]
[261] ../node_modules/firebase-admin/lib/auth/auth.js 20.5 KiB {0} [built]
[262] ../node_modules/@firebase/database/dist/index.esm.js 592 KiB {0} [built]
[264] ../node_modules/firebase-admin/lib/messaging/messaging.js 48.8 KiB {0} [built]
[265] ../node_modules/firebase-admin/lib/storage/storage.js 4.98 KiB {0} [built]
[354] ../node_modules/@google-cloud/firestore/build/src/index.js 52.9 KiB {0} [built]
[422] ./components/serviceAccountKey.json 2.32 KiB {0} [built]
[423] ./firebase.js 1.17 KiB {0} [built]
[424] ../node_modules/firebase-admin/lib/default-namespace.js 1.16 KiB {0} [built]
[425] ../node_modules/firebase-admin/lib/firebase-namespace.js 18.4 KiB {0} [built]
    + 780 hidden modules

WARNING in ../node_modules/grpc/src/grpc_extension.js 32:12-33
Critical dependency: the request of a dependency is an expression
 @ ../node_modules/grpc/index.js
 @ ../node_modules/google-gax/build/src/grpc.js
 @ ../node_modules/google-gax/build/src/index.js
 @ ../node_modules/@google-cloud/firestore/build/src/v1beta1/firestore_client.js
 @ ../node_modules/@google-cloud/firestore/build/src/v1beta1/index.js
 @ ../node_modules/@google-cloud/firestore/build/src/index.js
 @ ../node_modules/firebase-admin/lib/firebase-namespace.js
 @ ../node_modules/firebase-admin/lib/default-namespace.js
 @ ../node_modules/firebase-admin/lib/index.js
 @ ./firebase.js

WARNING in ../node_modules/grpc/node_modules/node-pre-gyp/lib/pre-binding.js 20:22-48
Critical dependency: the request of a dependency is an expression
 @ ../node_modules/grpc/src/grpc_extension.js
 @ ../node_modules/grpc/index.js
 @ ../node_modules/google-gax/build/src/grpc.js
 @ ../node_modules/google-gax/build/src/index.js
 @ ../node_modules/@google-cloud/firestore/build/src/v1beta1/firestore_client.js
 @ ../node_modules/@google-cloud/firestore/build/src/v1beta1/index.js
 @ ../node_modules/@google-cloud/firestore/build/src/index.js
 @ ../node_modules/firebase-admin/lib/firebase-namespace.js
 @ ../node_modules/firebase-admin/lib/default-namespace.js
 @ ../node_modules/firebase-admin/lib/index.js
 @ ./firebase.js

WARNING in ../node_modules/grpc/node_modules/node-pre-gyp/lib/util/versioning.js 17:20-67
Critical dependency: the request of a dependency is an expression
 @ ../node_modules/grpc/node_modules/node-pre-gyp/lib/pre-binding.js
 @ ../node_modules/grpc/src/grpc_extension.js
 @ ../node_modules/grpc/index.js
 @ ../node_modules/google-gax/build/src/grpc.js
 @ ../node_modules/google-gax/build/src/index.js
 @ ../node_modules/@google-cloud/firestore/build/src/v1beta1/firestore_client.js
 @ ../node_modules/@google-cloud/firestore/build/src/v1beta1/index.js
 @ ../node_modules/@google-cloud/firestore/build/src/index.js
 @ ../node_modules/firebase-admin/lib/firebase-namespace.js
 @ ../node_modules/firebase-admin/lib/default-namespace.js
 @ ../node_modules/firebase-admin/lib/index.js
 @ ./firebase.js

WARNING in ../node_modules/hash-stream-validation/index.js
Module not found: Error: Can't resolve 'fast-crc32c' in '/repos/test/node_modules/hash-stream-validation'
 @ ../node_modules/hash-stream-validation/index.js
 @ ../node_modules/@google-cloud/storage/build/src/file.js
 @ ../node_modules/@google-cloud/storage/build/src/index.js
 @ ../node_modules/firebase-admin/lib/storage/storage.js
 @ ../node_modules/firebase-admin/lib/firebase-namespace.js
 @ ../node_modules/firebase-admin/lib/default-namespace.js
 @ ../node_modules/firebase-admin/lib/index.js
 @ ./firebase.js

WARNING in ../node_modules/bytebuffer/dist/bytebuffer-node.js
Module not found: Error: Can't resolve 'memcpy' in '/repos/test/node_modules/bytebuffer/dist'
 @ ../node_modules/bytebuffer/dist/bytebuffer-node.js
 @ ../node_modules/protobufjs/dist/protobuf.js
 @ ../node_modules/grpc/index.js
 @ ../node_modules/google-gax/build/src/grpc.js
 @ ../node_modules/google-gax/build/src/index.js
 @ ../node_modules/@google-cloud/firestore/build/src/v1beta1/firestore_client.js
 @ ../node_modules/@google-cloud/firestore/build/src/v1beta1/index.js
 @ ../node_modules/@google-cloud/firestore/build/src/index.js
 @ ../node_modules/firebase-admin/lib/firebase-namespace.js
 @ ../node_modules/firebase-admin/lib/default-namespace.js
 @ ../node_modules/firebase-admin/lib/index.js
 @ ./firebase.js

WARNING in ../node_modules/write-file-atomic/index.js
Module not found: Error: Can't resolve 'worker_threads' in '/repos/test/node_modules/write-file-atomic'
 @ ../node_modules/write-file-atomic/index.js
 @ ../node_modules/gcs-resumable-upload/node_modules/configstore/index.js
 @ ../node_modules/gcs-resumable-upload/build/src/index.js
 @ ../node_modules/@google-cloud/storage/build/src/file.js
 @ ../node_modules/@google-cloud/storage/build/src/index.js
 @ ../node_modules/firebase-admin/lib/storage/storage.js
 @ ../node_modules/firebase-admin/lib/firebase-namespace.js
 @ ../node_modules/firebase-admin/lib/default-namespace.js
 @ ../node_modules/firebase-admin/lib/index.js
 @ ./firebase.js

Or using firebase-admin:

import * as admin from 'firebase-admin'

exports.handler = (event, context, callback) => callback(null, {
  statusCode: 200,
  body: 'OK'
})
Output:
netlify-lambda: Starting server
Lambda server is listening on 9000
Hash: e3817619e6c76f23fc78
Version: webpack 4.29.0
Time: 3102ms
Built at: 02/02/2019 11:53:21 PM
      Asset      Size  Chunks             Chunk Names
firebase.js  2.72 MiB       0  [emitted]  firebase
Entrypoint firebase = firebase.js
 [10] external "fs" 42 bytes {0} [built]
 [12] ../node_modules/firebase-admin/lib/utils/error.js 34.8 KiB {0} [built]
 [14] ../node_modules/firebase-admin/lib/utils/validator.js 7.25 KiB {0} [built]
 [56] ../node_modules/firebase-admin/lib/utils/deep-copy.js 2.6 KiB {0} [built]
[115] ../node_modules/firebase-admin/lib/auth/credential.js 15 KiB {0} [built]
[235] ../node_modules/firebase-admin/lib/index.js 1.36 KiB {0} [built]
[261] ../node_modules/firebase-admin/lib/auth/auth.js 20.5 KiB {0} [built]
[262] ../node_modules/@firebase/database/dist/index.esm.js 592 KiB {0} [built]
[264] ../node_modules/firebase-admin/lib/messaging/messaging.js 48.8 KiB {0} [built]
[265] ../node_modules/firebase-admin/lib/storage/storage.js 4.98 KiB {0} [built]
[354] ../node_modules/@google-cloud/firestore/build/src/index.js 52.9 KiB {0} [built]
[422] ./components/serviceAccountKey.json 2.32 KiB {0} [built]
[423] ./firebase.js 1.17 KiB {0} [built]
[424] ../node_modules/firebase-admin/lib/default-namespace.js 1.16 KiB {0} [built]
[425] ../node_modules/firebase-admin/lib/firebase-namespace.js 18.4 KiB {0} [built]
    + 780 hidden modules

WARNING in ../node_modules/grpc/src/grpc_extension.js 32:12-33
Critical dependency: the request of a dependency is an expression
 @ ../node_modules/grpc/index.js
 @ ../node_modules/google-gax/build/src/grpc.js
 @ ../node_modules/google-gax/build/src/index.js
 @ ../node_modules/@google-cloud/firestore/build/src/v1beta1/firestore_client.js
 @ ../node_modules/@google-cloud/firestore/build/src/v1beta1/index.js
 @ ../node_modules/@google-cloud/firestore/build/src/index.js
 @ ../node_modules/firebase-admin/lib/firebase-namespace.js
 @ ../node_modules/firebase-admin/lib/default-namespace.js
 @ ../node_modules/firebase-admin/lib/index.js
 @ ./firebase.js

WARNING in ../node_modules/grpc/node_modules/node-pre-gyp/lib/pre-binding.js 20:22-48
Critical dependency: the request of a dependency is an expression
 @ ../node_modules/grpc/src/grpc_extension.js
 @ ../node_modules/grpc/index.js
 @ ../node_modules/google-gax/build/src/grpc.js
 @ ../node_modules/google-gax/build/src/index.js
 @ ../node_modules/@google-cloud/firestore/build/src/v1beta1/firestore_client.js
 @ ../node_modules/@google-cloud/firestore/build/src/v1beta1/index.js
 @ ../node_modules/@google-cloud/firestore/build/src/index.js
 @ ../node_modules/firebase-admin/lib/firebase-namespace.js
 @ ../node_modules/firebase-admin/lib/default-namespace.js
 @ ../node_modules/firebase-admin/lib/index.js
 @ ./firebase.js

WARNING in ../node_modules/grpc/node_modules/node-pre-gyp/lib/util/versioning.js 17:20-67
Critical dependency: the request of a dependency is an expression
 @ ../node_modules/grpc/node_modules/node-pre-gyp/lib/pre-binding.js
 @ ../node_modules/grpc/src/grpc_extension.js
 @ ../node_modules/grpc/index.js
 @ ../node_modules/google-gax/build/src/grpc.js
 @ ../node_modules/google-gax/build/src/index.js
 @ ../node_modules/@google-cloud/firestore/build/src/v1beta1/firestore_client.js
 @ ../node_modules/@google-cloud/firestore/build/src/v1beta1/index.js
 @ ../node_modules/@google-cloud/firestore/build/src/index.js
 @ ../node_modules/firebase-admin/lib/firebase-namespace.js
 @ ../node_modules/firebase-admin/lib/default-namespace.js
 @ ../node_modules/firebase-admin/lib/index.js
 @ ./firebase.js

WARNING in ../node_modules/hash-stream-validation/index.js
Module not found: Error: Can't resolve 'fast-crc32c' in '/repos/test/node_modules/hash-stream-validation'
 @ ../node_modules/hash-stream-validation/index.js
 @ ../node_modules/@google-cloud/storage/build/src/file.js
 @ ../node_modules/@google-cloud/storage/build/src/index.js
 @ ../node_modules/firebase-admin/lib/storage/storage.js
 @ ../node_modules/firebase-admin/lib/firebase-namespace.js
 @ ../node_modules/firebase-admin/lib/default-namespace.js
 @ ../node_modules/firebase-admin/lib/index.js
 @ ./firebase.js

WARNING in ../node_modules/bytebuffer/dist/bytebuffer-node.js
Module not found: Error: Can't resolve 'memcpy' in '/repos/test/node_modules/bytebuffer/dist'
 @ ../node_modules/bytebuffer/dist/bytebuffer-node.js
 @ ../node_modules/protobufjs/dist/protobuf.js
 @ ../node_modules/grpc/index.js
 @ ../node_modules/google-gax/build/src/grpc.js
 @ ../node_modules/google-gax/build/src/index.js
 @ ../node_modules/@google-cloud/firestore/build/src/v1beta1/firestore_client.js
 @ ../node_modules/@google-cloud/firestore/build/src/v1beta1/index.js
 @ ../node_modules/@google-cloud/firestore/build/src/index.js
 @ ../node_modules/firebase-admin/lib/firebase-namespace.js
 @ ../node_modules/firebase-admin/lib/default-namespace.js
 @ ../node_modules/firebase-admin/lib/index.js
 @ ./firebase.js

WARNING in ../node_modules/write-file-atomic/index.js
Module not found: Error: Can't resolve 'worker_threads' in '/repos/test/node_modules/write-file-atomic'
 @ ../node_modules/write-file-atomic/index.js
 @ ../node_modules/gcs-resumable-upload/node_modules/configstore/index.js
 @ ../node_modules/gcs-resumable-upload/build/src/index.js
 @ ../node_modules/@google-cloud/storage/build/src/file.js
 @ ../node_modules/@google-cloud/storage/build/src/index.js
 @ ../node_modules/firebase-admin/lib/storage/storage.js
 @ ../node_modules/firebase-admin/lib/firebase-namespace.js
 @ ../node_modules/firebase-admin/lib/default-namespace.js
 @ ../node_modules/firebase-admin/lib/index.js
 @ ./firebase.js
@danew

This comment has been minimized.

Copy link

@danew danew commented Feb 3, 2019

I believe this is because that package does't like being bundled with netlify-lambda's default webpack config.
I'm currently also having the same issue, I'm hoping to have some time to investigate it further tomorrow.

@danew

This comment has been minimized.

Copy link

@danew danew commented Feb 3, 2019

It turns out that this issue is because the packages are being bundled with Webpack. Since the developer never intended this, since it's a "backend" module and wasn't intended to be bundled.

A fix for this would be to package your functions as a zip file like @DavidWells has done here https://github.com/DavidWells/function-zips/

@sw-yx

This comment has been minimized.

Copy link
Contributor

@sw-yx sw-yx commented Feb 3, 2019

i dont have experience with firebase sdk but @danew makes sense. zip it up. i'd also welcome a PR for instructions on zipping.

we will be releasing more native support for this kind of thing in a couple months - please stay tuned.

@sw-yx

This comment has been minimized.

Copy link
Contributor

@sw-yx sw-yx commented Feb 3, 2019

@rick-stevens

This comment has been minimized.

Copy link
Author

@rick-stevens rick-stevens commented Feb 3, 2019

Thanks @danew / @sw-yx! This workaround does, however, cut out netlify-lambda, which was the reason for this issue. There's no need to zip the Firebase SDK if I'm not using the transpiler in the first place (as also shown in the blog post you linked from the README).

It makes developing harder as I'll have to write my own file watcher and proxy server to integrate with my app's framework. Looking forward to the native support!

@sw-yx

This comment has been minimized.

Copy link
Contributor

@sw-yx sw-yx commented Feb 3, 2019

ok, i see. i love using firebase too and am keen to provide better DX for this. let me know if you figure out a way.

@danew

This comment has been minimized.

Copy link

@danew danew commented Feb 3, 2019

@rick-stevens as an alternative you can still use netlify-lambda serve for local development by using a custom webpack config which includes webpack-node-externals. This will stop webpack from bundling your bundles these packages together and require them from your node_modules instead.

Then you just need to create a simple build script similar to the one I linked above for build time.

@chinchang

This comment has been minimized.

Copy link

@chinchang chinchang commented Feb 14, 2019

Even "zip n ship" is not a solution. node_modules installed on user's machine won't necessarily work on Netlify function server. grpc has the same issues. I am guessing it gets compiled according to the host machine architecture.

@sw-yx

This comment has been minimized.

Copy link
Contributor

@sw-yx sw-yx commented Feb 15, 2019

hmm. if you can make a small repro i can take it to our platform devs. i would v much like to support this usecase in production (even without netlify-lambda)

@sw-yx

This comment has been minimized.

Copy link
Contributor

@sw-yx sw-yx commented Feb 15, 2019

@rick-stevens just noting a similar stackoverflow answer https://stackoverflow.com/questions/51522280/use-firebase-sdk-with-netlify-lambda-functions/51656271 it seems that instead of using the SDK you can just use their REST endpoint?

@rick-stevens

This comment has been minimized.

Copy link
Author

@rick-stevens rick-stevens commented Feb 15, 2019

Hi @sw-yx, I think this is the same Stack Overflow issue I noted originally. I already tried looking into their REST API, but while this is okay for some of their services, it doesn't provide one for Firebase Storage for instance (one I need in my case).

I've already switched to a full Firebase stack for my project (including Hosting and Functions), so I don't have a quick example anymore. But including firebase-admin (or other libraries like grpc as @chinchang stated) into an empty netlify-lambda project function, should be sufficient to reproduce!

@sw-yx

This comment has been minimized.

Copy link
Contributor

@sw-yx sw-yx commented Feb 15, 2019

my bad, i skipped right past that SO answer. i see, its firebase storage thats needed. good enough for us to work with, thank you!

@chinchang

This comment has been minimized.

Copy link

@chinchang chinchang commented Feb 15, 2019

I too switched to Firebase functions. And wanted to highlight that their functions deployment experience is much better than netlify as there is no need of packaging nor uploading node_modules. It just works. Maybe we can get a similar experience in Netlify also :)

@thomaswangio

This comment has been minimized.

Copy link

@thomaswangio thomaswangio commented Feb 15, 2019

hey @sw-yx , yea also the REST solution does not work for Cloud Firestore, only for the Realtime Database. An integrated netlify-lambda solution would be much appreciated.

@renatodex

This comment has been minimized.

Copy link

@renatodex renatodex commented Mar 29, 2019

+1000000 here.

This issue is massive for us.
We had to basically to implement the Whole Firebase SDK in Rest to allow this integration.
Of course we love Netlify Lambda functions much more than Firebase functions, since we are hosting the project on Netlify, and we don't want to lose the awesome Deploy features provided by Netlify.

The firebase authentication using Service Accounts is still a huge problem for us, since there is not enough documentation to clarify this process without using Google SDKs.

I opened a thread on firebase-admin also to grab more attention to this.
firebase/firebase-admin-node#491

@mklan

This comment has been minimized.

Copy link

@mklan mklan commented May 2, 2019

Hey do you still have the issue? I found a reasonable solution:

add this webpack configuration to your project

//./config/webpack.functions.js
const nodeExternals = require('webpack-node-externals');

module.exports = {
  externals: [nodeExternals()],
};

and update your scripts inside your package.json to take a custom webpack config:

"scripts": {
    "build:lambda": "netlify-lambda build --config ./config/webpack.functions.js src/lambda",
    "start:lambda": "netlify-lambda serve --config ./config/webpack.functions.js src/lambda"
  },
@rick-stevens

This comment has been minimized.

Copy link
Author

@rick-stevens rick-stevens commented May 2, 2019

Hi @mklan, I've personally moved away from this approach all together, but by looking at the thread activity I think other people will probably have a go at this. Thanks!

@sw-yx

This comment has been minimized.

Copy link
Contributor

@sw-yx sw-yx commented May 2, 2019

thats awesome @mklan! do you think it makes sense to build it into netlify-lambda?

@mklan

This comment has been minimized.

Copy link

@mklan mklan commented May 3, 2019

@sw-yx I am not sure what side effects this config has.

@sw-yx

This comment has been minimized.

Copy link
Contributor

@sw-yx sw-yx commented May 3, 2019

ok. cc @renatodex

@juanojeda

This comment has been minimized.

Copy link

@juanojeda juanojeda commented May 16, 2019

//./config/webpack.functions.js
const nodeExternals = require('webpack-node-externals');

module.exports = {
  externals: [nodeExternals()],
};

and update your scripts inside your package.json to take a custom webpack config:

"scripts": {
    "build:lambda": "netlify-lambda build --config ./config/webpack.functions.js src/lambda",
    "start:lambda": "netlify-lambda serve --config ./config/webpack.functions.js src/lambda"
  },

I've been banging my head against this issue today, and finally came across this solution. It works for me, and it'd be awesome if it was documented in netlify-lambda somewhere.

👍 Thanks gang!

@sw-yx

This comment has been minimized.

Copy link
Contributor

@sw-yx sw-yx commented May 16, 2019

@apotox

This comment has been minimized.

Copy link

@apotox apotox commented Jun 27, 2019

in my case when I tried to bundle firebase SDK with my lambda function I faced a lot of errors and missed requirements.., so decided to move the big external dependencies with new package.json file inside “lambda” folder with the help of generate-json-webpack-plugin webpack plugin. and I edited the package.json build script , so after the “yarn build: server” I change the directory by the command “cd dist/server/” and then “yarn install”, this will create a new node_modules into the lambda folder and netlify will zip it using zip-it-and-ship-it to AWS lambda function.

Check This Medium Post

@stale

This comment has been minimized.

Copy link

@stale stale bot commented Aug 26, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.