Allow development of Firebase Cloud Functions #836
Comments
|
I am interested in this but I am not sure what all I would want on this to make it work easier. |
|
I would like to see it easier to build out the functions project while referencing libraries. Here are some other ideas:
I think that is it off the top of my head. |
|
I think we could divide it into milestones:
Can we agree on that? Did I miss something? |
|
Yeah I think that is perfect |
|
@FrozenPandaz I think this feature has gained quite a popularity. I do realize that there may be other, more important tasks on hand, but we would like to get an update if there are even plans to take it on. |
|
If anyone is using firebase in a nx managed monorepo, you should be aware of this behavior: firebase/firebase-js-sdk#1696 (comment) |
|
@Bielik20 Setting "source": "/" in your firebase.json is not ideal at all as 'firebase deploy' will then try to upload all your source code. I've been banging my head against a wall try to get this to work. Ideally 'source' would be something like 'dist/apps/functions'. |
|
@jongood01 Yeah I agree but then firebase would complain that there is no npm package there: It is annoying but I think there is little harm in keeping source code up there. It cannot be accessed by anyone from the outside. Deployment is a little longer but also nothing one should worry about. Or is there a specific reason against it you have in mind? |
|
@Bielik20 No other reason but on our team we run the deploy as part of a CI process and the mono repo is going to get huge with a lot of code that has nothing to do with the code on Firebase so builds will get slower and slower. I had a hack around to create a minimal package.json in the dist folder but this was also a hack around and not ideal either. Will pick one or the other option for now. |
|
Am I right - this issue affects deployment of Nest.js app as well? Btw, any movements on this issue? |
|
Hi @Bielik20 and @jongood01 I had also run into the problem of not wanting to upload all my source code. What i found is that you can configure functions to ignore certain directories in firebase.json like this: Documentation here: https://firebase.google.com/docs/hosting/full-config |
|
If nrwl could work natively with firebase cloud functions, that would be ideal. I'm building an app with Firebase. The amount of hours I've spent fixing configuration errors and trying to get local emulation and testing to work is far higher than time spent doing actual programming. Nothing works, ever. I've already tried Lerna but it seems to be more-so designed for open source projects. Not to mention that its package hoisting breaks everything, meaning I need to npm install each package manually anyway, defeating the purpose. I was hoping that nrwl would finally help in this regard. A setup that would allow me to start actually programming again rather than adjusting config files all day. |
|
Guys, I've found a workaround, that allows using single TLDR: https://github.com/spy4x/nx-with-firebase-functions Idea: Firebase Functions need only 2 files to work:
So we can use I created a simple JS script that gets a list of appName-specific dependencies from the And it works fine! You can run The only flaw of this workaround that I see now - it's necessary to manually update root Check my demo repo (link above) for details. I split changes into commits, so you can see that there is no magic behind. The main change is in this commit: https://github.com/spy4x/nx-with-firebase-functions/commit/c95997976df1f985c2fce146708cb26ace3f5208 I hope that helps somebody. And maybe someone will find a way to update P.S. you can use any triggers, dependencies and frameworks (ie Nest.js) with this approach. Just don't forget to update |
|
Google Cloud Functions Generator Generate a Google Cloud Function within a Nx workspace with dev tools:
I took Nx development strategy for front-end components & applied it to back-end microservices (Google Cloud Functions). With my plugin, I can create & deploy production-ready microservices in 5 minutes. I then combine my microservices to develop a business automation strategy, business analytics, or data streaming pipeline. |
|
@spy4x That's amazing! Thank you for sharing! We have started to provide the ability for the community to provide plugins for Nx and I think this would be a great candidate. If @spy4x or somebody else would be interested in maintaining a plugin for Nx + Firebase functions integrations, we would be happy to help and promote it! |
|
My take on @spy4x's solution above. I added the This means that there's no more need for a separate const packageJson = require('../../package.json') // Take root package.json
const path = require('path')
const fs = require('fs').promises
const depcheck = require('depcheck')
const ROOT_PATH = path.resolve(__dirname + '/../..')
const distProjectPath = `${ROOT_PATH}/dist/apps/functions`
console.log('Creating cloud functions package.json file...')
let packageJsonStub = {
engines: { node: '10' },
main: 'main.js',
}
depcheck(
distProjectPath,
{
package: {
dependencies: packageJson.dependencies,
},
},
unused => {
let dependencies = packageJson.dependencies
if (unused.dependencies.length > 0)
console.log('Deleting unused dependencies:')
unused.dependencies.reduce((acc, dep, i) => {
console.log(`${i + 1} - Deleting ${dep}`)
delete acc[dep]
return acc
}, dependencies)
fs.mkdir(path.dirname(distProjectPath), { recursive: true }).then(() => {
fs.writeFile(
`${distProjectPath}/package.json`,
JSON.stringify({
...packageJsonStub,
dependencies,
})
)
.then(() =>
console.log(`${distProjectPath}/package.json written successfully.`)
)
.catch(e => console.error(e))
})
}
) |
|
@jaytavares Great job! I have one more improvement for this setup. Instead of custom scripts in package.json, we could add builders with {
"pkgJson": {
"builder": "@nrwl/workspace:run-commands",
"options": {
"command": "ts-node scripts/build-firebase-package-json.ts",
"cwd": "tools"
}
},
"serve": {
"builder": "@nrwl/workspace:run-commands",
"options": {
"command": "firebase emulators:start --only functions --inspect-functions"
}
},
"shell": {
"builder": "@nrwl/workspace:run-commands",
"options": {
"command": "firebase functions:shell --inspect-functions"
}
},
"deploy": {
"builder": "@nrwl/workspace:run-commands",
"options": {
"command": "firebase deploy"
}
}
} |
|
@vdjurdjevic 👍 That's the same approach I used to be able to use "architect": {
"build-node": {
"builder": "@nrwl/node:build",
"options": { },
"configurations": { }
},
"build": {
"builder": "@nrwl/workspace:run-commands",
"options": {
"commands": [
{
"command": "nx run functions:build-node"
},
{
"command": "node tools/scripts/build-cloud-functions-package-file.js"
}
],
"parallel": false
}, |
|
@vdjurdjevic @jaytavares Thanks for the snippets. I was able to get Firebase Functions up and running with NestJS within a Nx workspace. However, I need to run PS: Not sure if it's worth mentioning but initially I have |
|
Watch works. Just run |
|
I opened an issue in firebase repo to support a more flexible project structure for functions. That would enable better integration with Nx, and I would be willing to contribute full-featured plugin for firebase (not just functions, but firestore config, hosting, storage, etc..). @FrozenPandaz can you please check it out and tell me what you think? |
|
I created an angular workspace and then I wrapped my functions under a node app but I'm struggling to use imported libraries since I'm getting this error when trying to deploy firebase functions: Error: Cannot find module '@nrwl/workspace' Anyone faced this? Any help is appreciated. |
|
Hello I have the same issue.... any news about that?
|
|
I found out that this is related to the One quick way to make it work is removing the postinstall script from package.json before deploying to functions, then add it back again so the nx client doesn't break. Is there a way to skip the postinstall script when deploying to firebase functions? |
You're probably looking for the
|
|
Just to add an additional tweak to @dobromyslov's awesome comment: The angular.json Note: I was also having an issue where it was saying "build": {
"builder": "@nrwl/workspace:run-commands",
"options": {
"commands": [
{
"command": "nx run functions:build-node"
},
{
"command": "npx ts-node tools/scripts/build-firebase-functions-package-json.ts"
}
],
"parallel": false
},
"configurations": {
"production": {
"prod": true
}
}
}Specifying the |
|
There is this community plugin, not sure how well it works: https://github.com/JoelCode/gcp-function |
|
@markwhitfeld thank you for the addition. I also did that but did not post here.
gcp-function plugin does not provide full flexibility as such as you develop a standalone Firebase project. Standard Firebase approach lets you mix multiple functions as well as mix various triggers (e.g. firestore document change, storage file write, scheduler) and create a full featured data processing pipeline. |
|
Does anybody know how to properly create and manage 2 or more Firebase projects under one NX workspace? I am struggling one drawback of @spy4x @jaytavares @vdjurdjevic approach: can't split dependencies between functions. I added one function which requires puppetteer package. As you may know this dependency is rather heavy and takes much time during functions deployment. My idea concludes in dependencies separation between functions or even better between several Firebase projects:
|
|
Found a way to create, run local emulators and deploy two or more NX apps with Firebase support using Just add second NX application with Firebase support as described above: #836 (comment) and add another Expand firebase.second-app.jsonAnd use |
I think this is the proper way of doing things within a monorepo & cloud functions. Firebase CFs are powerfull, but in monorepos, we have quite a lot of code. Only one node application (i.e. So, I see What are your best pratices guys ? |
|
Now that NX node build supports generatePackageJson (https://nx.dev/latest/angular/node/build#generatepackagejson) does this negate the need for the separate build-firebase-functions-package-json script? |
Yes, that's what it means for me at least. I was just able to test the {
"engines": {
"node": "12"
},
"main": "main.js"
}Big 👍 to the Nx team for their implementation of the |
|
@IainAdamsLabs thank you for the hint. Seems like this tool simplifies boilerplate. |
|
Am I the one here who doesn't see the provided solution with webpack good enough? Firebase Functions processed with webpack potentially have bad outcome. With only one function it's good, but with more functions the webpack solution doesn't seem pretty sweet. At least the provided solution doesn't fit my use case.
I rolled up a fully custom solution which uses just typescript compiler (no webpack) with opportunity to import shared local libraries. Final functions folder structure for Firestore structure: Sample import * as functions from 'firebase-functions';
import * as admin from 'firebase-admin';
import { User } from '@WORKSPACE/shared/types';
const firestore = admin.firestore();
export default functions
.firestore.document('profile/{profileId}')
.onCreate(async (snapshot, context) => {
// ...
// const u: User = { ... };
}As you may notice, folder structure reflects Firestore structure. All functions to be deployed are suffixed with With given Deployed functions names will reflect camelCased folder structure. Deployed With below files you will be able to achieve that structure with NX + have an ability to import local Source code filesapps/functions/src/main.tsExpandimport './pathAliases';
import * as admin from 'firebase-admin';
import glob from 'glob';
import camelCase from 'camelcase';
import { resolve } from 'path';
admin.initializeApp();
// add more folders for function detection if needed
const lookupFolders = ['firestore'];
const baseFoldersRegex = `(${lookupFolders.join('|')})`;
const EXTENSION = '.f.js';
const files = glob.sync(`./+${baseFoldersRegex}/**/*${EXTENSION}`, {
cwd: __dirname
});
for (let f = 0, fl = files.length; f < fl; f++) {
const file = files[f];
const functionName = camelCase(
file
.split(EXTENSION)
.join('')
.split('/')
.join('_')
.replace(new RegExp(baseFoldersRegex), '')
);
if (
!process.env.FUNCTION_NAME ||
process.env.FUNCTION_NAME === functionName
) {
// eslint-disable-next-line
const mod = require(resolve(__dirname, file));
exports[functionName] = mod.default || mod;
}
}apps/functions/src/pathAliases.tsExpandimport ModuleAlias from 'module-alias';
import path from 'path';
import { getAliasesForSharedLibs } from './sharedLibs';
// paths in tsconfig.base.ts will let typescript recognize the paths
// paths specified and registerd by module-alias will register paths for
// compiled javascript
const paths = {
...getAliasesForSharedLibs()
};
const aliases = Object.entries(paths).reduce((acc, [alias, paths]) => {
const aliasCorrected = alias.replace('/*', '');
const pp = paths[0].replace('/*', '/');
const pathCorrected = path
.join(__dirname, pp)
.replace(/\\/g, '/')
.replace('.ts', '.js');
return {
...acc,
[aliasCorrected]: pathCorrected
};
}, {});
ModuleAlias.addAliases(aliases);apps/functions/src/sharedLibs.tsExpandexport const sharedLibsConfig = {
/**
* List of all libs in the monorepo that functions import
*/
libs: ['shared/types'],
/**
* Path to a folder in which output all shared libraries
*
* @note Folder is relative to functions root folder
*/
outDir: 'libs'
};
export function getAliasesForSharedLibs(): Record<string, string[]> {
const WORKSPACE_NAME = 'PUT_YOUR_WORKSPACE_NAME_HERE';
const { libs, outDir } = sharedLibsConfig;
return libs.reduce((result, libName) => {
const alias = `@${WORKSPACE_NAME}/${libName}`;
const paths = [`../${outDir}/${libName}/src/index.js`];
return { ...result, [alias]: paths };
}, {});
}You have to only adjust
apps/functions/tsconfig.app.jsonExpand{
"extends": "./tsconfig.json",
"compilerOptions": {
"outDir": "../../dist",
"sourceMap": false,
"module": "commonjs",
"types": ["node"],
"strict": true,
"removeComments": true,
"esModuleInterop": true
},
"exclude": ["**/*.spec.ts"],
"include": ["**/*.ts"]
}Make sure that Workspace configurationfirebase.jsonExpand{
"functions": {
"source": "dist/apps/functions",
"predeploy": ["nx run functions:build"],
"runtime": "nodejs10"
}
}tools/scripts/copy-shared-libs-for-functions.tsExpandimport * as path from 'path';
import * as fs from 'fs-extra';
async function main() {
const args = process.argv.slice(2);
if (!args?.length || !args[0]) {
throw new Error('Application name must be provided.');
}
const APP_NAME = args[0];
const CFG_FILE = 'sharedLibs.js'; // matches (functions/src/sharedLibs.ts)
const ROOT_PATH = path.resolve(__dirname + '/../..');
const DIST_FOLDER = `${ROOT_PATH}/dist`;
const { sharedLibsConfig } = await import(
`${DIST_FOLDER}/apps/${APP_NAME}/src/${CFG_FILE}`
);
if (!sharedLibsConfig?.libs || !sharedLibsConfig?.outDir) {
throw new Error('Config for copying shared libs has wrong format.');
}
const { libs, outDir } = sharedLibsConfig;
console.log('Copying shared libraries:', libs);
console.log('Total:', libs.length);
console.log(
'Destination:',
`${DIST_FOLDER}/apps/${APP_NAME}/${sharedLibsConfig.outDir}`
);
const promises = libs.map(async (libName: string) => {
const src = `${DIST_FOLDER}/libs/${libName}`;
const dest = path.join(DIST_FOLDER, `apps/${APP_NAME}`, outDir, libName);
await fs.copy(src, dest, {
overwrite: true
});
});
await Promise.all(promises);
}
main()
.then(() => {
// Nothing to do
})
.catch((error) => {
console.error(error);
process.exit(1);
});tools/scripts/build-firebase-functions-package-json.tsExpandimport * as path from 'path';
import * as depcheck from 'depcheck';
import * as fs from 'fs';
const PACKAGE_JSON_TEMPLATE = {
engines: { node: '10' },
main: 'src/main.js'
};
async function main(): Promise<void> {
const args = process.argv.slice(2);
if (!args?.length || !args[0]) {
throw new Error('Application name must be provided.');
}
const APPLICATION_NAME = args[0];
/*****************************************************************************
* package.json
* - Filter unused dependencies.
* - Write custom package.json to the dist directory.
****************************************************************************/
const ROOT_PATH = path.resolve(__dirname + '/../..');
const DIST_PROJECT_PATH = `${ROOT_PATH}/dist/apps/${APPLICATION_NAME}`;
const packageJson = require('../../package.json');
console.log('Creating cloud functions package.json file...');
// Get unused dependencies
const { dependencies: unusedDependencies } = await depcheck(
DIST_PROJECT_PATH,
{
package: {
dependencies: packageJson.dependencies
}
}
);
// Filter dependencies
const requiredDependencies = Object.entries(
packageJson.dependencies as { [key: string]: string }
)
?.filter(([key, value]) => !unusedDependencies?.includes(key))
?.reduce<{ [key: string]: string }>((previousValue, [key, value]) => {
previousValue[key] = value;
return previousValue;
}, {});
console.log(
`Required dependencies count: ${
Object.values(requiredDependencies)?.length
}`
);
// Write custom package.json to the dist directory
await fs.promises.mkdir(path.dirname(DIST_PROJECT_PATH), { recursive: true });
await fs.promises.writeFile(
`${DIST_PROJECT_PATH}/package.json`,
JSON.stringify(
{
...PACKAGE_JSON_TEMPLATE,
dependencies: requiredDependencies
},
undefined,
2
)
);
console.log(`Written successfully: ${DIST_PROJECT_PATH}/package.json`);
}
main()
.then(() => {
// Nothing to do
})
.catch((error) => {
console.error(error);
process.exit(1);
});Root package.jsonExpand "dependencies": {
"camelcase": "^6.2.0",
"fs-extra": "^9.1.0",
"glob": "^7.1.6",
"module-alias": "^2.2.2",
// ...
},
"devDependencies": {
"@types/fs-extra": "^9.0.7",
"@types/glob": "^7.1.3",
"@types/module-alias": "^2.0.0",
"copyfiles": "^2.4.1",
"depcheck": "^1.4.0",
"onchange": "^7.1.0",
"rimraf": "^3.0.2",
"ts-node": "~9.1.1",
"typescript": "4.0.2",
"wait-on": "^5.2.1",
// ...
},workspace.jsonExpand "functions": {
"root": "apps/functions",
"sourceRoot": "apps/functions/src",
"projectType": "application",
"prefix": "functions",
"targets": {
"compile": {
"executor": "@nrwl/workspace:run-commands",
"options": {
"commands": [
"npx tsc -p apps/functions/tsconfig.app.json",
"npx ts-node tools/scripts/copy-shared-libs-for-functions.ts functions",
"npx ts-node tools/scripts/build-firebase-functions-package-json.ts functions"
],
"parallel": false
}
},
"build": {
"executor": "@nrwl/workspace:run-commands",
"options": {
"commands": [
"npx rimraf dist/apps/functions",
"nx compile functions",
"cd dist/apps/functions && npm install --package-lock-only"
],
"parallel": false
}
},
"compile-dev-assets": {
"executor": "@nrwl/workspace:run-commands",
"options": {
"commands": [
"npx onchange -ik libs/**/*.ts -- npx ts-node tools/scripts/copy-shared-libs-for-functions.ts functions",
"npx ts-node tools/scripts/build-firebase-functions-package-json.ts functions",
"npx copyfiles apps/functions/.runtimeconfig.json dist"
]
}
},
"serve": {
"executor": "@nrwl/workspace:run-commands",
"options": {
"commands": [
"npx tsc -p apps/functions/tsconfig.app.json -w",
"npx wait-on dist/apps/functions && nx compile-dev-assets functions",
"npx wait-on dist/apps/functions/package.json && firebase serve --only functions"
]
}
},
"shell": {
"executor": "@nrwl/workspace:run-commands",
"options": {
"command": "nx build-dev functions && firebase functions:shell --inspect-functions"
}
},
"download-runtimeconfig": {
"executor": "@nrwl/workspace:run-commands",
"options": {
"commands": [
"firebase functions:config:get > apps/functions/.runtimeconfig.json"
]
}
},
"logs": {
"executor": "@nrwl/workspace:run-commands",
"options": {
"command": "firebase functions:log"
}
},
"lint": {
"executor": "@nrwl/linter:eslint",
"options": {
"lintFilePatterns": ["apps/functions/**/*.ts"]
}
},
"test": {
"executor": "@nrwl/jest:jest",
"outputs": ["coverage/apps/functions"],
"options": {
"jestConfig": "apps/functions/jest.config.js",
"passWithNoTests": true
}
}
}
},
How this worksWith given workspace structure: let's assume we have to share types between functions and webapp (i.e. nextjs): import { User } from '@WORKSPACE/shared/types';For webapp no need for any action. For functions to import shared libs, you will have to add them to // sharedLibs.ts
export const sharedLibsConfig = {
libs: ['shared/types'], // <-- here
outDir: 'libs'
};That's it. After that running If you look inside the
The power of module-alias and just copying files around is a very strong combo :) So far this solution has been working great for me. I hope this will be useful for somebody too! |
|
A |
|
@IainAdamsLabs @jaytavares I tested Then I compared package.json generated by |
|
This issue has been automatically marked as stale because it hasn't had any recent activity. It will be closed in 14 days if no further activity occurs. |
|
What is the advantage of setting all of this without webpack (@shelooks16) compared to this alternative: https://itnext.io/nx-nest-firebase-the-dream-616e8ee71920 Also the pros of getting rid of webpack to achieve "automatic function exports" is a big plus to me but I am not sure if the NestJS alternative does that as well. |
|
@KingDarBoja Hi! I do not use Nest.js. And this is not what was initially desired. What about other function types, triggers, pubsub? Nest solution creates one function of type onRequest to serve the API. It is good but serves completely different purpose. The solution I posted above allows to develop functions in the same way as without NX, but at the same it allows to share libs. And of course "automatic function exports" is yum 😃 And to be honest, I think Nest.js is not that good for serverless environments since it has a pretty good amount of boot time which increases the cold start. |
|
@shelooks16 would you mind sharing project structure ? thx a lot :) |
|
@maeri Hi. Yeah, sure :) At very basic extent, we have something like this (all 3 apps share types, dates, currency libs). We did not split functions app itself into smaller libs: There is also a separate script that starts firebase emulators (if needed). Above solution uses the outdated way, here is what we do right now: |
|
Hi folks, I too was looking for a Firebase solution for my own use with Nx, and was following this thread with interest, so I decided to have a bash at a custom plugin, and after a couple of weeks figuring out how plugins (and Nx) work , I have published a first version to npm. If anyone wants to give it a try out I'd be glad to hear if its useful to others (I'm currently using it in my own monorepo). My approach was to try and integrate Firebase workflow within Nx in only a lightly opinionated way that gives us the ability to use Nx libraries in firebase functions code but still be able to use the firebase CLI and also other Firebase features (hosting, rules, indexes, etc.) within Nx without any fuss. The plugin uses a custom builder to compile functions code as a plain Typescript library, and then do all of the post-build work to auto-generate dependency outputs in dist that are ready to deploy. Nx Library support works by copying build output for each dependent Nx library to the output folder and referencing them in the functions package.json as local modules. |
|
I did a small try with that plugin locally in some experimental repo and seems like that's what everyone been looking for. Followed the steps and got into the build part and the output api lib does seems to retain the structure of automatic function exports, although I haven't tested sharing libs between this firebase app and other libs. Looking for others to give it a try, awesome work @simondotm ❤️ EDIT Well, I am happy with the output result, seems like grouped exports are kept by using TSC and importing node libs does what is mentioned in the readme. App Folder Structure Output Structure Build log $ nx build coljobs-api --with-deps
> NX Running target build for project coljobs-api and 1 task(s) that it depends on.
———————————————————————————————————————————————
> nx run coljobs-api:build [retrieved from cache]
Compiling TypeScript files for project "coljobs-api"...
Done compiling TypeScript files for project "coljobs-api".
Copying asset files...
Done copying asset files.
- Processing dependencies for firebase functions app
- Firebase functions app has 'npm' dependency 'firebase-functions'
- Firebase functions app has 'lib' dependency '@ngfire-showcase/company/domain'
- Updating firebase package.json
———————————————————————————————————————————————
> NX SUCCESS Running target "build" succeeded
Nx read the output from cache instead of running the command for 2 out of 2 tasks. |
|
Hello again folks, I've just released v0.3.0 of my plugin, which now has some improvements and fixes, as well as support for |
|
Firebase is getting more popular by the day. It certainly deserves to be a first-class Nx workspace citizen. It's a shame that the Nx team decided to outsource it to community plugins. |
|
@paxforce As much as we would all love to have first class support for firebase, we need to remember that it adds a lot of maintenance work on Nx team. I feel like it is better for them to focus on improving Nx itself and community tools. There is tremendous amount of use cases for monorepo. Multiple languages, frameworks, tools. I kind of agree that from their perspective it is better to outsource that work to the community, since for example they may not have a firebase expert on hand to create that implementation. |
|
@shelooks16 Is this structure even suitable for Angular 12 with Nx Workspace? |
Yeah, why not? Structure of one NX app does not affect other NX apps. The suggested functions approach is complicated but it works. If you need help with setting it up, I will be glad to help :) As an alternative, checkout this package which achieves similar result (posted above): |
|
This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context. |
and others
Cloud Functions are a great way of developing a micro-service based backend. Just like with node backend it is common to share types and some business logic between the two. Could we come up with some way of integrating them with an Nx Workspace? What are the things that people would like to see in that area?
Current Behavior
No easy way of integrating a Cloud Functions within an Nx Workspace.
Expected Behavior
Developers should be able to easily develop a Cloud Functions backend within an Nx Workspace.
The text was updated successfully, but these errors were encountered: