Launch endpoints refactor #3831
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Fork translator implementations * TriggerDiscovery now records Endpoints as well as Specs * Remove debug print
* Fork translator implementations * TriggerDiscovery now records Endpoints as well as Specs * Remove debug print * Start moving over existing use cases
* Fork translator implementations * TriggerDiscovery now records Endpoints as well as Specs * Remove debug print * Start moving over existing use cases * Migrate the deploy codebase aside from release code * PR feedback
This change introduces the refactor discussed in go/emulator-multi-tenancy. Main changes include: * Updating how `ProjectState` is indexed in `getProjectStateById()` * Extending `ProjectState` with `AgentProjectState` and `TenantProjectState` Corresponding internal bug: b/192387245
* Update API schema from discovery. * WIP: Add MFA support. * Implement withdraw and token fields. * Add error handling for most cases. * Clean up some comments. * Support MFA in batchCreate. * Properly support MFA for emailLink and IDP. * Address review feedback. * Clean up more TODOs.
Add preprocessing logic to use flatPaths instead of paths in discovery object. Some endpoints/methods have the same path, which led to endpoints being overwritten and not appearing in apiSpec.js (e.g. see identitytoolkit.projects.inboundSamlConfigs.delete and identitytoolkit.projects.defaultSupportedIdpConfigs.delete in v2 discovery docs). To work around this, use the flatPath which are distinct between overloaded paths. This change largely impacts paths that have params with slashes in them, which is currently not supported by OpenAPI. Corresponding internal bug: b/199768026
* Adding noninteractive support to ext:dev:publish * Add --non-interactive and --force support to ext:update (#3749) * ext:update now supports --non-interactive and --force * fixed test * add changelog * formats
…r Runtime. (#3707) As a step towards standardizing Functions Runtime environment, we extract all logic for initializing environment variables for a n emulated function out of the `FunctionsEmulatorRuntime` and place them in the caller (`FunctionsEmulator`). All behaviors should remain the same minus the following details: 1. Environment variable values will not change once the runtime has started. Env vars like `FUNCTIONS_TARGET` might look like it should change per function invocation, but since each worker is associated with a trigger,`FUNCTION_TARGET` won't actually ever change. This may be a problem though if other emulators (e.g. storage emulator) is started _after_ the functions emulator is started and a function is invoked. (I don't think this is a common use case?). 2. We no longer check user's Functions SDK version to optionally override realtime emulator path. I think it's actually better to fail the emulator IF user has configured realtime database emulator and using an unsupported version of the functions SDK rather than silently talking to the prod database. I can be convinced otherwise though.
https://snyk.io/vuln/SNYK-JS-PACRESOLVER-1564857 firebase-tools@9.17.0 requires pac-resolver@^4.1.0 via a transitive dependency on pac-proxy-agent@4.1.0. upgrading to proxy-agent@5.0.0 causes pac-resolver to get upgraded to 5.0.0, resolving the issue. Co-authored-by: Bryan Kendall <bkend@google.com>
* Minor fixes for Auth Emulator IDP widget. * Changelog.
* migrating parseRef to ref * moving ref parsing into its own file * add jsdocs * renaming to refs * forgot to save
Update PubSub Emulator to emit events in CloudEvent format given a trigger whose function signature is `cloudevent`. Previously, the Functions Emulator considered 2 types of functions - `https` and `background`. With the introduction of [CloudEvents](https://cloud.google.com/functions/docs/writing/cloudevents) in Google Cloud Functions, we now have 2 types of `background` functions - `legacy` vs `cloudevent`. For example: ```js /* Legacy Events */ exports.helloPubSub = functions.pubsub.topic('topic-name').onPublish((message) => { // ... }); /* CloudEvents */ exports.helloPubSub = functions.v2.pubsub.topic('topic-name').onPublish((cloudevent) => { const message = cloudevent.data.message; }); ``` To accommodate, we adopt concept of "function signature" introduced in Google Cloud Function's [Functions Framework](https://github.com/GoogleCloudPlatform/functions-framework-nodejs) to categorize each function trigger as one of `http`, `event`, or`cloudevent`. When registering a function trigger with the Pubsub emulator, the functions's signature type will be included in the registration request, and the Pubsub emulator will use the info to emit an event in the format expected by the underlying function trigger. We also make minor changes to the Functions Emulator to selectively massage the incoming events in-route to the function trigger. As an added bonus, we are now able to remove `triggerType` field in the `FunctionsRuntimeArg` as that info is now outdated and not needed when invoking an emulated function.
* Fork translator implementations * TriggerDiscovery now records Endpoints as well as Specs * Remove debug print * Start moving over existing use cases * Migrate the deploy codebase aside from release code * Migrate deploymentPlanner * Make code easier to review * PR feedback
* Start inferring wantBackend details from existingBackend * Update src/test/deploy/functions/prepare.spec.ts Co-authored-by: Daniel Lee <danielylee@google.com> Co-authored-by: Daniel Lee <danielylee@google.com>
* Adds Fabricator to release code Fabricator is the repalcement for the old Tasks codebase, though it will not handle the portion of tracking or error reporting internally (those will be future utilities for better testing). Some differences between tasks.ts and fabricator.ts: * Queues are used for narrow transactions, which means we don't accidentally have an error bubble up and cause a retry on a bulk operation. There's a design pattern of executor.run.catch(rethrowAs) * Function triggers (not EventArc triggers, which are deployed as part of the function, but external triggers like topics, and in the future intents or task queues) are only created after a successful function creation and functions are only deleted after a successful trigger deletion * We do not delete functions if an upsert failed (this helps catch cases where a user renamed a function) * This code is actually unit tested (hooray!) * Delete scratch space code * PR feedback + tests & fixes for SourceTokenScraper
Merge master; backport @colerogers' changes to Endpoint code.
Adds Fabricator to release code Fabricator is the repalcement for the old Tasks codebase, though it will not handle the portion of tracking or error reporting internally (those will be future utilities for better testing). Some differences between tasks.ts and fabricator.ts: * Queues are used for narrow transactions, which means we don't accidentally have an error bubble up and cause a retry on a bulk operation. There's a design pattern of executor.run.catch(rethrowAs) * Function triggers (not EventArc triggers, which are deployed as part of the function, but external triggers like topics, and in the future intents or task queues) are only created after a successful function creation and functions are only deleted after a successful trigger deletion * We do not delete functions if an upsert failed (this helps catch cases where a user renamed a function) * This code is actually unit tested (hooray!)
* Move functions:delete to endpoints code * Remove old functions delete code
* Make container contract parsing use Endpoints * Finish the endpoints refactor! (#3827) * Delete dead code; make a few fixes as I discover them * Free up 12LOC to meet budget
|
All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter. We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the ℹ️ Googlers: Go here for more info. |
* Don't obscure helpful error messages * Functions should have deployment-tool label and functions:delete shouldn't need it * Lost commit? Record URI after deploys for future printing * Lint fixes * Alternative deploymentTool fix * get rid of extra space in logOpStart * Don't cache trigger regions if the trigger changed * functions list should list ID not entrypoint * Fix compiler erro * Rewrite trigger region copy code * Enablement of APIs should have 'functions' prefix * Remember old trigger region. Recreate Fn when it changes * Run formatter
|
All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter. We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the ℹ️ Googlers: Go here for more info. |
inlined
added
cla: yes
|
All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter. We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the ℹ️ Googlers: Go here for more info. |
google-cla
bot
added
cla: no
and removed
cla: yes
|
This is entirely a composition of previously LGTM'd commits and merges from master. It should be pretty rubber stamp ready |
inlined
added
cla: yes
devpeerapong
pushed a commit
to devpeerapong/firebase-tools
that referenced
this pull request
Dec 14, 2021
* Trigger discovery now includes Endpoints (firebase#3752) * Fork translator implementations * TriggerDiscovery now records Endpoints as well as Specs * Remove debug print * Start moving over cases of existingBackend (firebase#3753) * Fork translator implementations * TriggerDiscovery now records Endpoints as well as Specs * Remove debug print * Start moving over existing use cases * Migrate the deploy codebase aside from release code (firebase#3754) * Fork translator implementations * TriggerDiscovery now records Endpoints as well as Specs * Remove debug print * Start moving over existing use cases * Migrate the deploy codebase aside from release code * PR feedback * Use eventType to infer service from function trigger definition in the Emulator. (firebase#3712) * Refactor in preparation for adding multitenancy support (firebase#3746) This change introduces the refactor discussed in go/emulator-multi-tenancy. Main changes include: * Updating how `ProjectState` is indexed in `getProjectStateById()` * Extending `ProjectState` with `AgentProjectState` and `TenantProjectState` Corresponding internal bug: b/192387245 * Add MFA support for Auth Emulator (firebase#3732) * Update API schema from discovery. * WIP: Add MFA support. * Implement withdraw and token fields. * Add error handling for most cases. * Clean up some comments. * Support MFA in batchCreate. * Properly support MFA for emailLink and IDP. * Address review feedback. * Clean up more TODOs. * Update gen-auth-api-spec.ts to use flatPath instead of path (firebase#3761) Add preprocessing logic to use flatPaths instead of paths in discovery object. Some endpoints/methods have the same path, which led to endpoints being overwritten and not appearing in apiSpec.js (e.g. see identitytoolkit.projects.inboundSamlConfigs.delete and identitytoolkit.projects.defaultSupportedIdpConfigs.delete in v2 discovery docs). To work around this, use the flatPath which are distinct between overloaded paths. This change largely impacts paths that have params with slashes in them, which is currently not supported by OpenAPI. Corresponding internal bug: b/199768026 * Adding noninteractive support to ext:dev:publish (firebase#3745) * Adding noninteractive support to ext:dev:publish * Add --non-interactive and --force support to ext:update (firebase#3749) * ext:update now supports --non-interactive and --force * fixed test * add changelog * formats * Extract environment variables initialization out of Functions Emulator Runtime. (firebase#3707) As a step towards standardizing Functions Runtime environment, we extract all logic for initializing environment variables for a n emulated function out of the `FunctionsEmulatorRuntime` and place them in the caller (`FunctionsEmulator`). All behaviors should remain the same minus the following details: 1. Environment variable values will not change once the runtime has started. Env vars like `FUNCTIONS_TARGET` might look like it should change per function invocation, but since each worker is associated with a trigger,`FUNCTION_TARGET` won't actually ever change. This may be a problem though if other emulators (e.g. storage emulator) is started _after_ the functions emulator is started and a function is invoked. (I don't think this is a common use case?). 2. We no longer check user's Functions SDK version to optionally override realtime emulator path. I think it's actually better to fail the emulator IF user has configured realtime database emulator and using an unsupported version of the functions SDK rather than silently talking to the prod database. I can be convinced otherwise though. * security(deps): proxy-agent@5.0.0 (CVE-2021-23406) (firebase#3757) https://snyk.io/vuln/SNYK-JS-PACRESOLVER-1564857 firebase-tools@9.17.0 requires pac-resolver@^4.1.0 via a transitive dependency on pac-proxy-agent@4.1.0. upgrading to proxy-agent@5.0.0 causes pac-resolver to get upgraded to 5.0.0, resolving the issue. Co-authored-by: Bryan Kendall <bkend@google.com> * adding externalServices to extensions types (firebase#3766) * fix login:use and account resolution issue (firebase#3773) * Minor fixes for Auth Emulator IDP widget. (firebase#3774) * Minor fixes for Auth Emulator IDP widget. * Changelog. * Move ref parsing code to a separate file. (firebase#3779) * migrating parseRef to ref * moving ref parsing into its own file * add jsdocs * renaming to refs * forgot to save * Generate CloudEvents in the PubSub Emulator (firebase#3767) Update PubSub Emulator to emit events in CloudEvent format given a trigger whose function signature is `cloudevent`. Previously, the Functions Emulator considered 2 types of functions - `https` and `background`. With the introduction of [CloudEvents](https://cloud.google.com/functions/docs/writing/cloudevents) in Google Cloud Functions, we now have 2 types of `background` functions - `legacy` vs `cloudevent`. For example: ```js /* Legacy Events */ exports.helloPubSub = functions.pubsub.topic('topic-name').onPublish((message) => { // ... }); /* CloudEvents */ exports.helloPubSub = functions.v2.pubsub.topic('topic-name').onPublish((cloudevent) => { const message = cloudevent.data.message; }); ``` To accommodate, we adopt concept of "function signature" introduced in Google Cloud Function's [Functions Framework](https://github.com/GoogleCloudPlatform/functions-framework-nodejs) to categorize each function trigger as one of `http`, `event`, or`cloudevent`. When registering a function trigger with the Pubsub emulator, the functions's signature type will be included in the registration request, and the Pubsub emulator will use the info to emit an event in the format expected by the underlying function trigger. We also make minor changes to the Functions Emulator to selectively massage the incoming events in-route to the function trigger. As an added bonus, we are now able to remove `triggerType` field in the `FunctionsRuntimeArg` as that info is now outdated and not needed when invoking an emulated function. * Add Endpoints version of DeploymentPlanner (firebase#3756) * Fork translator implementations * TriggerDiscovery now records Endpoints as well as Specs * Remove debug print * Start moving over existing use cases * Migrate the deploy codebase aside from release code * Migrate deploymentPlanner * Make code easier to review * PR feedback * Start inferring wantBackend details from existingBackend (firebase#3790) * Start inferring wantBackend details from existingBackend * Update src/test/deploy/functions/prepare.spec.ts Co-authored-by: Daniel Lee <danielylee@google.com> Co-authored-by: Daniel Lee <danielylee@google.com> * Adds Fabricator to release code (firebase#3807) * Adds Fabricator to release code Fabricator is the repalcement for the old Tasks codebase, though it will not handle the portion of tracking or error reporting internally (those will be future utilities for better testing). Some differences between tasks.ts and fabricator.ts: * Queues are used for narrow transactions, which means we don't accidentally have an error bubble up and cause a retry on a bulk operation. There's a design pattern of executor.run.catch(rethrowAs) * Function triggers (not EventArc triggers, which are deployed as part of the function, but external triggers like topics, and in the future intents or task queues) are only created after a successful function creation and functions are only deleted after a successful trigger deletion * We do not delete functions if an upsert failed (this helps catch cases where a user renamed a function) * This code is actually unit tested (hooray!) * Delete scratch space code * PR feedback + tests & fixes for SourceTokenScraper * Merge master; backport recent changes (firebase#3816) Merge master; backport @colerogers' changes to Endpoint code. * Add metrics and error reporting to release code (firebase#3815) Adds Fabricator to release code Fabricator is the repalcement for the old Tasks codebase, though it will not handle the portion of tracking or error reporting internally (those will be future utilities for better testing). Some differences between tasks.ts and fabricator.ts: * Queues are used for narrow transactions, which means we don't accidentally have an error bubble up and cause a retry on a bulk operation. There's a design pattern of executor.run.catch(rethrowAs) * Function triggers (not EventArc triggers, which are deployed as part of the function, but external triggers like topics, and in the future intents or task queues) are only created after a successful function creation and functions are only deleted after a successful trigger deletion * We do not delete functions if an upsert failed (this helps catch cases where a user renamed a function) * This code is actually unit tested (hooray!) * Move firebase functions:delete to use Endpoints code (firebase#3821) * Move functions:delete to endpoints code * Remove old functions delete code * Make container contract parsing use Endpoints (firebase#3826) * Inlined.endpoints 9.6.discovery (firebase#3830) * Make container contract parsing use Endpoints * Finish the endpoints refactor! (firebase#3827) * Delete dead code; make a few fixes as I discover them * Free up 12LOC to meet budget * Fixes from bugbashing (firebase#3832) * Don't obscure helpful error messages * Functions should have deployment-tool label and functions:delete shouldn't need it * Lost commit? Record URI after deploys for future printing * Lint fixes * Alternative deploymentTool fix * get rid of extra space in logOpStart * Don't cache trigger regions if the trigger changed * functions list should list ID not entrypoint * Fix compiler erro * Rewrite trigger region copy code * Enablement of APIs should have 'functions' prefix * Remember old trigger region. Recreate Fn when it changes * Run formatter Co-authored-by: Daniel Lee <danielylee@google.com> Co-authored-by: Lisa Jian <lisajian@google.com> Co-authored-by: Yuchen Shi <yuchenshi@google.com> Co-authored-by: joehan <joehanley@google.com> Co-authored-by: Avi Vahl <avi.vahl@wix.com> Co-authored-by: Bryan Kendall <bkend@google.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Aggregated changes for the Endpoints refactor. This change replaces FunctionSpec/PubSubSpec/ScheduleSpc with Endpoint. Endpoint is also loosely what we parse now for the v1alpha1 container contract, though we use a slightly different type (no
idorprojectand 1region1 is an optional array).Some product changes:
backend.Triggeredto include a new type union)