Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-Language Strategy and Interoperability #2430

Open
swgillespie opened this Issue Feb 5, 2019 · 6 comments

Comments

Projects
None yet
5 participants
@swgillespie
Copy link
Member

swgillespie commented Feb 5, 2019

Today, Pulumi is officially capable of hosting programs written in JavaScript/TypeScript, Python, and Go. It is also capable of running programs written in C# and F#. Pulumi advertises itself as a multi-language system and, indeed, we have spent a lot of engineering time and effort ensuring that Pulumi supports more than one language.

Supporting multiple languages is an adoption boon. Pulumi itself is a new programming model and, if a user does not already know the language that they are using with Pulumi, it becomes a hugely daunting task to learn both a new language and Pulumi's programming model simultaneously. If Pulumi supports a language that a user already knows, it becomes much easier to understand how Pulumi works and how to use it effectively.

However, we have found that the maintenance cost of multiple language frontends for Pulumi is very high. We have put a large amount of engineering effort towards bringing our Python support to feature-parity with NodeJS, as @Frassle also has done for .NET. As it stands today it is difficult for us as a small group of engineers to scale effectively when our efforts must be spread out and duplicated across languages.

Furthermore, Pulumi users and those of us working at Pulumi alike have built a number of useful high-level libraries. (1 2 3 4). These libraries encapsulate into reusable components tasks that are otherwise hard to do, such as creating EKS clusters, creating AWS best-practice network topologies, and orchestrating Docker builds. It is clear that part of Pulumi's "magic" is the ability to author reusable components in a real programming language and be able to publish them through normal code push mechanisms such as NPM or PyPI.

Having said all of this, a clear contradiction in the "Pulumi story" arises: reusable components can only be used if the component is implemented in the same language that a user is using with Pulumi. A user today writing Pulumi programs in Python can't use any of the above linked packages and must re-write them wholesale in order to get any value from them. We are acutely aware of this issue.

Current State

Today, Pulumi supports a few languages, but none of the libraries written in those languages can be reused unless you're using the same language. Because the Pulumi team has invested a large majority of their engineering time and effort in TypeScript, all of our high-level libraries are written in TypeScript and can only be consumed by Pulumi programs written in TypeScript or JavaScript. Pulumi programs written in Python or Go have essentially zero libraries.

We do not believe that the "status quo" is sustainable at its current rate. We do not have the engineering bandwidth to manually duplicate all high-level libraries that we provide across all supported languages. Even if we did, the combinatorial explosion of language-library pairs makes it harder and harder to add additional languages to Pulumi as Pulumi's ecosystem as a whole grows.

Short-term Plan

In the M21 timeframe (the next 5 weeks), we plan on pursuing a prototype for "multi-language interop" - the ability for a component authored in one language to be used naturally in another language. The goal of this prototype is to write a Pulumi program in Python that creates an EKS cluster using the @pulumi/eks package, which is itself authored in TypeScript. This is an R&D effort. The goal here is to investigate what the prototype solution looks like and how it fits into the rest of the Pulumi ecosystem.

In the short-term, we are not going to put any engineering effort into Go language support for Pulumi. The hope is that, by pursuing a prototype, we will unlock a path to multi-language interop that will reduce the cost of adding additional languages to Pulumi so that we can once more accept additional languages into the Pulumi ecosystem.

Long-term Plan

At the end of the M21 timeframe, we will evaluate the R&D effort and make a decision about Pulumi's multi-language story. The various axes that we will consider include:

  1. The prevalence of new component libraries and their usage
  2. Whether or not the prototype accomplished its primary goal of interoperating with @pulumi/eks
  3. Whether or not the prototype accomplished its secondary goals of producing a general system and scheme for interoperating languages and libraries
  4. The estimated time cost of productizing the prototype based on our experience creating it

The are multiple options on the table with regards to Pulumi's multi-language story going forward, which we must decide on:

  1. "Status quo". After pursuing the prototype and finding it lacking or infeasible for whatever reason, we will continue as we are today, supporting multiple languages that do not interoperate with one another. We will do the work to re-write commonly used component libaries in languages that need them, driven by user demand.
  2. Productize the multi-language interop prototype with the goal of authoring a set of component libraries in a single language that can be used from multiple languages.
  3. Drop support for languages other than JavaScript/TypeScript.

Prototype Plan

In M21, @swgillespie will pursue a prototype that allows language hosts to launch other language hosts, remote objects across process boundaries, and launch instances of libraries in those child language hosts. The goal of the prototype is to allow for the invoking of constructors and the invoking of functions across language boundaries. We have put a lot of thought into candidate designs for a prototype and this design is the one that has shown the most promise.

The goal of the prototype is to instantiate a @pulumi/eks Cluster resource in a Python program. This involves creating a way for the Python language host to launch a NodeJS language host, load the @pulumi/eks library in the target language host, pass arguments to the Cluster constructor, and receive outputs from the created Cluster object. The goal is explicitly not to create a general-purpose remote object protocol.

If the prototype is successful and we choose to productize it, the goal will be to produce a way for us as Pulumi engineers to author libraries that can be used in languages other than the implementation language. As it matures, the interop system may also be usable by Pulumi users so that they can author their own components.

@swgillespie swgillespie added this to the 0.21 milestone Feb 5, 2019

@swgillespie swgillespie self-assigned this Feb 5, 2019

@RobJellinghaus

This comment has been minimized.

Copy link

RobJellinghaus commented Feb 6, 2019

I know statically typed languages aren't really in your wheelhouse atm, but I couldn't help noticing the similarity to xlang.

Components are written using the type system of the implementation language and accessed using the type system of the consuming language. As such, the xlang type system is designed to interoperate with modern object-oriented programming languages. This includes both static (such as C#) and dynamic (such as Python) object-oriented languages as well as strongly typed (such as C++) and weakly typed (such as JavaScript) languages.

@pchalamet

This comment has been minimized.

Copy link

pchalamet commented Feb 6, 2019

Couldn't you just consider terraform as an internal pivot format for infrastructure description on which all lang could interrop? Did a POC last year (https://github.com/pchalamet/NTerraform): that was based on terraform providers to generate a lang bridge. Idea was to implement a type provider then in f# above but never really had time to go beyond this after vacations. Authoring experience is far better when you are taken by hand in the IDE and it's far better than yaml.
I do not know inner internals of pulumi but I guess it used to be bootstrapped like this at some point. Just speculating.

But yes, in contrast, this requires a lot of code generators and requires to extract metadata from terraform providers. But this can be automated and made available to most lang and looks like it's no big deal - it's just a matter of build steps and caching I would say (not saying it's easy). Drawbacks are you can't natively interrop with functions as pulumi requires it. So couldn't functions be pushed to well, just functions (azure functions for eg) or a docker? Pulumi could take care of all this plumbing with much benefits for users.

I do no have much to bring to the table, sorry. I'm just curious about your prototype and how you will solve this!

@swgillespie

This comment has been minimized.

Copy link
Member Author

swgillespie commented Feb 6, 2019

I know statically typed languages aren't really in your wheelhouse atm, but I couldn't help noticing the similarity to xlang.

xlang is a general-purpose remote object protocol, which (while useful) is far more powerful than we need and is generally counter to our goals:

The goal is explicitly not to create a general-purpose remote object protocol.

xlang (and it's predecessors, WinRT and COM) are very heavyweight. An explicit design goal is to produce something that is as lightweight as possible.

Couldn't you just consider terraform as an internal pivot format for infrastructure description on which all lang could interrop?

We do not want to tie ourselves to Terraform in the long run. We already do not use Terraform for our Kubernetes provider. In general, the fact that Pulumi uses Terraform providers at all is an implementation detail.

But yes, in contrast, this requires a lot of code generators and requires to extract metadata from terraform providers. But this can be automated and made available to most lang and looks like it's no big deal - it's just a matter of build steps and caching I would say (not saying it's easy).

We already do this: https://github.com/pulumi/pulumi-terraform. We have a tool called tfgen that extracts metadata from Terraform providers and generates code from them. You're right that this already "solves" the problem and, in fact, it's really the crux of how we've even gotten this far in the first place. However, it also intimately chains us to Terraform in a way that is not desirable.

@Frassle

This comment has been minimized.

Copy link

Frassle commented Feb 20, 2019

Is there a branch we can watch to get an idea of how this is coming along?

@swgillespie

This comment has been minimized.

Copy link
Member Author

swgillespie commented Feb 27, 2019

@Frassle not yet, but stay tuned!

@swgillespie

This comment has been minimized.

Copy link
Member Author

swgillespie commented Mar 9, 2019

Hi all, here's an update:

Our M21 timeframe is wrapping up today, and with it the R&D experiment that I mentioned in the OP. The primary goal of this exercise was to produce a prototype that was able to, from within a program written in Python, interoperate with the @pulumi/eks JavaScript package - and from that perspective, the experiment was mostly successful. The code for this is in another repository that isn't public for purely logistical reasons - I expect that as we proceed in M22 this code will either be open sourced directly or integrated into this repo.

The next step here is to move on to the decision-making portion of the task that I pointed out in the OP. In the M22 timeframe we're going to take a look at where we ended up and make a judgement call about what to do next. The plan hasn't changed since I wrote the OP.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.