Add new command (functions:secrets:set) for creating secrets to be used for CF3. #4021
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…nto dl-cf3-secrets
inlined
approved these changes
Feb 1, 2022
…ebase-tools into dl-cf3-secrets-cmds
taeold
added a commit
that referenced
this pull request
Feb 3, 2022
Follow up #4021 to add other management commands for CF3 secrets. Note that `destroy` commands can be improved by making sure we don't accidentally delete secrets versions currently in use (which would immediately break the function!). I'll add these feature in a follow up PR when we finish reviewing the PR w/ `prune` command.
taeold
added a commit
that referenced
this pull request
Feb 10, 2022
* Use runtime delegate to parse function triggers in the Functions Emulator (#4012) Today, Functions Emulator parses function trigger from source as follows: 1) Spin up an instance of Functions runtime process 2) Invoke a "function" which triggers a path that parses the trigger by calling out to `extractTriggers.js` 3) Send parsed triggers via IPC from runtime to emulator. Emulator now knows about the triggers. This has the advantage of running the trigger parsing in the emulated runtime (which properly mocks out calls to the DB, applies network filtering, uses the same node version when possible, etc.) but has the disadvantage of complicating the runtime implementation as well as diverging from how the triggers are parsed in `firebase deploy`. Using runtime delegate, we have: 1) Use runtime delegate to discover the delegate appropriate for function source (i.e. Node, but in the future can be some other runtime) 2) Spin up a node subprocess to parse trigger. Emulator now knows about the triggers. i.e. the same procedure used during `firebase deploy` By using runtime delegate, we align the function deploy to production and to the emulated environment and simplify the runtime code a bit. This also puts us into a good position in the future when we make the function deploy process a little more complex, e.g. params and secrets support. * Slim down Functions Emulator Runtime (i.e. args sent over to emulated functions) (#4105) Now that we've simplified the Functions Emulator to separate out process for loading triggers from the one running the trigger, we can slim down [`FunctionsRuntimeBundle`](https://github.com/firebase/firebase-tools/blob/2e68803f994dbe4f72eb0965dd6a12e7a043b597/src/emulator/functionsEmulatorShared.ts#L53-L88) that is passed between the Functions Emulator and the Functions Runtime process. This change removes almost all payload attributes in the Functions Runtime Bundle except `proto`. This is nice - we are getting very close to the payload that's passed to a production function instance. We have to leave couple of things like socketpath and debug features - this will probably be removed when we move over to pure-http based protocol (socket) and SDK based debug feature enablement. This could be worked later when I have a little more time! One more change - we pass around the whole trigger definition in the Functions Emulator instead of pieces of it. This makes it easier to do something else I'm doing... (secret emulator) in the subsequent PR. * CF3 Secrets Support (#3959) Support deploying secret environment variables on a function. Prior to deploying functions with secret configuration, the CLI will run somewhat comprehensive validation to ensure that secret config will work when deployed, e.g. 1. Secret version exists. 2. Secret version is in ENABLED state. 3. Secret version can be access by the runtime service account. We do this since the GCF doesn't do the same level of validation and instead repeatedly fail to spin up a new instance with an invalid secret config. This often results on super long deploys (probably until some master timeout is met for function instance deploy). I took the opportunity to refactor the code a little to group various "ensure" and "validate" used in function deploys in their own files. Emulator support for secrets will come in a separate PR. * Add new command (functions:secrets:set) for creating secrets to be used for CF3. (#4021) One of several family of commands to be implemented for managing secrets for CF3. `functions:secrets:set` command is used to create a new secret version in Secret Manager. If a secret doesn't exist, a secret will be created before adding a new version. To guide users to our recommended best practices, we will only allow users to create secrets in `UPPER_SNAKE_CASE` - this makes it more obvious how these secrets can be accessed at runtime (via environment variable of the same name). Usage: ``` $ echo SHHHH > SECRET_FILE $ firebase functions:secrets:set MY_SECRET --data-file=SECRET_FILE ✔ Created a new secret version projects/my-project/secrets/MY_SECRET/versions/0 i Please deploy your functions for the change to take effect by running: firebase deploy --only functions // Calling set on existing secret name will create a new version. $ echo SHHHHHHHH > SECRET_FILEE $ firebase functions:secrets:set MY_SECRET --data-file=SECRET_FILE ✔ Created a new secret version projects/my-project/secrets/MY_SECRET/versions/1 i Please deploy your functions for the change to take effect by running firebase deploy --only functions // "-" as STDIN is supported but discouraged since it will leave the secret in shell history $ echo SHHHHHHHHHH | firebase functions:secrets:set --data-file=- MY_SECRET ✔ Created a new secret version projects/my-project/secrets/MY_SECRET/versions/2 i Please deploy your functions for the change to take effect by running firebase deploy --only functions // Without --data-file flag, begin interactive prompt to take user input $ firebase functions:secrets:set MY_SECRET ? Enter a value for MY_SECRET [input is hidden]: ✔ Created a new secret version projects/my-project/secrets/MY_SECRET/versions/3 i Please deploy your functions for the change to take effect by running: firebase deploy --only functions ``` * Add functions:secrets:{access, destroy, get} commands. (#4026) Follow up #4021 to add other management commands for CF3 secrets. Note that `destroy` commands can be improved by making sure we don't accidentally delete secrets versions currently in use (which would immediately break the function!). I'll add these feature in a follow up PR when we finish reviewing the PR w/ `prune` command. * Add command to prune unused secrets (#4108) Each active secret version cost money. To help save cost on using Secret Manager, we add `functions:secrets:prune` command which: 1) Looks up all secret versions from secrets marked with label "firebase-managed". All secrets created using the Firebase CLI will have this label. 2) Look up all secret bindings for CF3 function instance. 3) Figure out which secret version isn't currently being used. Since destroying a secret version is irrevocable and immediately breaking for clients that depend on it, we will always ask for a confirmation for the destroy operations (and not support -f flag). Note that we now query `v1` of Secret Manager since `v1beta` does not offer filtering by labels. * Add support for secrets in the Functions Emulator (#4106) Emulator will now recognize function triggers with secret environments and ensure that secret environment variables are populated in the emulated runtime. Secrets in Functions Emulator can come from 2 sources: 1) From local override file (`.secret.local`). 2) From Google Cloud Secret Manager. In this case, default application credentials (i.e. credentials used in Firebase CLI) will be used to fetch the secret from GCP. As suspected, (1) take precedence over (2). If accessing secret from GCP fails for any reason, the Emulator logs, but does not throw, the failed attempt and proceeds to execute the functions code. Some refactoring changes needed to be in the Emulator: * Some functions turned into async. * We pass around the whole trigger in more places. * Remove preview flag, add option to disable dotenv support. (#4022) Preparing for launching dotenv support for CF3. At launch, CF3 environment variables support will default to picking up dotenv file if any, without need for preview flag. * Add EXT_ as a reserved environment variable prefix (#4148) Firebase Extensions use environment variables with `EXT_` for many of their "first class" environment variable keys. We will add it to the reserved prefix list before releasing the dotenv support for all users.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
One of several family of commands to be implemented for managing secrets for CF3.
functions:secrets:setcommand is used to create a new secret version in Secret Manager. If a secret doesn't exist, a secret will be created before adding a new version.To guide users to our recommended best practices, we will only allow users to create secrets in
UPPER_SNAKE_CASE- this makes it more obvious how these secrets can be accessed at runtime (via environment variable of the same name).Usage:
This PR depends on #3959.