Version providers #2389
Comments
lukehoban
added
the
p1
|
Marking as P1 in M22 as this is a critical part of being able to adopt breaking changes in providers, which may be coming from upstream for several of our core providers. |
|
Strange, I thought we already did this. (I definitely recall very technically deep discussions with Sean and/or Pat about how this would work in terms of RPC and checkpoint file changes, Fall of last year.) Did we just design, but not implement, it? |
|
@swgillespie Is there anything more planned for this in M21 after merging #2483? Could we update this issue with the steps taken so far and thoughts on what we might want to do as additional future investments? (Or close this out and open new issues?) |
|
So far, we have taken #2483, which contains a fix for what was the arguably bug-like behavior of plugin loading: the plugin loader would load plugins that were "newer" than the one requested, which posed a problem if the newer plugin was not compatible with the requested version. For M21, this was deemed sufficiently troublesome that we took a fix for it. Entering M22, the current state of provider versioning is that the CLI will load exactly the version of the plugin requested by a Pulumi program, and will error if that specific plugin is not available. This should alleviate the immediate pain of some incoming breaking changes we are planning to absorb in some downstream providers, so that this issue is no longer "on fire" and blocking the uptake of new providers. For M22, we should pursue a more holistic solution to the problem. We discussed a number of options that we can take here and, although it's not clear what the best solution is at this point, from my perspective the one that seemed most promising was to have Pulumi resources directly inform the engine of the provider version upon resource registration (concretely, provider packages would embed their version into their source code and pass it to the engine during In the early M22 timeframe, we'll take a look at our options and understand the technical blockers in place. For the above solution, we will at minimum need to augment the |
|
See pulumi/pulumi-gcp#115 for a case where users hit issues due to having two copies of Handling this correctly will require support for SxS plugins and much of the additional work outlined in #2389 (comment). |
|
A related issue, which cause serious problems, described here: https://pulumi-community.slack.com/archives/C84L4E3N1/p1554209058169500 |
|
Even after cleaning out plugins, I'm seeing incorrect older gcp plugins reappear as documented here pulumi/pulumi-gcp#124 (comment) It seems I cannot workaround the issue. |
In pursuit of #2389, this commit adds the necessary changes to the resource monitor protocol so that language hosts can communicate exactly what version of a provider should be used when servicing an Invoke, ReadResource, or RegisterResource. The expectation here is that, if a language host provides a version, the engine MUST use EXACTLY that version of a provider plugin in order to service the request.
In pursuit of #2389, this commit adds the necessary changes to the resource monitor protocol so that language hosts can communicate exactly what version of a provider should be used when servicing an Invoke, ReadResource, or RegisterResource. The expectation here is that, if a language host provides a version, the engine MUST use EXACTLY that version of a provider plugin in order to service the request.
In pursuit of #2389, this commit adds the necessary changes to the resource monitor protocol so that language hosts can communicate exactly what version of a provider should be used when servicing an Invoke, ReadResource, or RegisterResource. The expectation here is that, if a language host provides a version, the engine MUST use EXACTLY that version of a provider plugin in order to service the request.
As part of #2389, we need the ability for language hosts to tell the engine that a particular resource registration, read, or invoke needs to use a particular version of a resource provider. This was not previously possible before; the engine prior to this commit loaded plugins from a default provider map, which was inferred for every resource provider based on the contents of a user's package.json, and was itself prone to bugs. This PR adds the engine support needed for language hosts to request a particular version of a provider. If this occurs, the source evaluator specifically records the intent to load a provider with a given version and produces a "default" provider registration that requests exactly that version. This allows the source evaluator to produce multiple default providers for a signle package, which was previously not possible. This is accomplished by having the source evaluator deal in the "ProviderRequest" type, which is a tuple of version and package. A request to load a provider whose version matches the package of a previously loaded provider will re-use the existing default provider. If the version was not previously loaded, a new default provider is injected.
* Load specific provider versions if requested As part of #2389, we need the ability for language hosts to tell the engine that a particular resource registration, read, or invoke needs to use a particular version of a resource provider. This was not previously possible before; the engine prior to this commit loaded plugins from a default provider map, which was inferred for every resource provider based on the contents of a user's package.json, and was itself prone to bugs. This PR adds the engine support needed for language hosts to request a particular version of a provider. If this occurs, the source evaluator specifically records the intent to load a provider with a given version and produces a "default" provider registration that requests exactly that version. This allows the source evaluator to produce multiple default providers for a signle package, which was previously not possible. This is accomplished by having the source evaluator deal in the "ProviderRequest" type, which is a tuple of version and package. A request to load a provider whose version matches the package of a previously loaded provider will re-use the existing default provider. If the version was not previously loaded, a new default provider is injected. * CR Feedback: raise error if semver is invalid * CR: call String() if you want a hash key * Update pkg/resource/deploy/providers/provider.go Co-Authored-By: swgillespie <sean@pulumi.com>
|
We got the core of the engine versioning implemented in M22. We still need to wire this through into the various providers in M23. |
|
We have completed the work here. When we did so, we discovered an issue which is tracked by #2753, but we'll use that that track the completion of this feature. |
Today, we always load the latest version available on the machine of the requested resource provider. However, the resource provider itself can be loaded by packages at:
This leads to problems if we ever make a (intentional or not) breaking change in a resource provider, as some of the code in the program may only work correctly with the version of the provider it was authored against.
We will need to find a way to ensure that we load the specific version(s) required by the program/packages. This will likely also require that we have the ability to load multiple versions of a provider into the same deployment, and to route resource operations from the program to the right versions of the providers.
Work items:
The text was updated successfully, but these errors were encountered: