Adds remote .zip archive support to `pulumi new` #14443

kpitzen · 2023-10-30T16:43:54Z

Description

Fixes https://github.com/pulumi/pulumi.ai/issues/363

This change introduces a new accepted suffix to pulumi new HTTP arguments - .zip.

When a URL of the form http[s]://<url>.zip is provided to pulumi new, we send a GET request to <url> (without the .zip suffix) with the accept header set to application/zip. It is then up to whatever is hosted at that location to correctly return a binary zip archive in its response.

We then parse that response in memory and write each file to a tempdir as we do with current templates, which can then be picked up by the standard pulumi new templating logic.

Checklist

I have run make tidy to update any new dependencies
I have run make lint to verify my code passes the lint check
- I have formatted my code using gofumpt

I have added tests that prove my fix is effective or that my feature works

I have run make changelog and committed the changelog/pending/<file> documenting my change

Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version

pulumi-bot · 2023-10-30T16:45:19Z

Changelog

[uncommitted] (2023-10-31)

Features

[cli/new] Adds support for remote zip archive templates to pulumi new
#14443

kpitzen · 2023-10-30T20:04:47Z

Added no-changelog-required per @AaronFriel

sdk/go/common/workspace/templates_ai.go

lukehoban · 2023-10-31T02:44:36Z

My suggested approach for this feature is a bit different from what is implemented here.

Add support for a general Zip format for Pulumi templates in the CLI. This is an alternative to acquiring via a git ref.
Have the AI service use that to provide a dynamically built Zip file as a source.

This avoids coupling details of AI to the Pulumi new feature, and maintains more flexibility to iterate and improve on the Pulumi AI server side, while exposing a more generally useful and foundational capability in the CLI.

Thoughts?

kpitzen · 2023-10-31T02:50:55Z

My suggested approach for this feature is a bit different from what is implemented here.
1. Add support for a general Zip format for Pulumi templates in the CLI. This is an alternative to acquiring via a git ref.

2. Have the AI service use that to provide a dynamically built Zip file as a source.
This avoids coupling details of AI to the Pulumi new feature, and maintains more flexibility to iterate and improve on the Pulumi AI server side, while exposing a more generally useful and foundational capability in the CLI.

Thoughts?

I considered this approach initially, but wanted to avoid building an API that would be difficult to iterate upon and test - providing the data in a structured format as I've done here has made it much simpler to work on prompt engineering, etc, while (with @Frassle 's prompting) keeping the logic in the CLI as simple as possible - we just need a list of filenames and contents which the CLI writes in the same place it would expect a standard template to exist. Notably, this does not couple the CLI to "AI" other than nominally in this case. We could just as easily rename all of these concepts and consider this a "REST standard" for Pulumi templates.

I suppose we could add a layer which, rather than returning the JSON we see here, returns the same contents as an archive, but it does make the API a bit more opaque (and any contracts provided by the service more difficult to enforce). I'd personally prefer to stick to structured data where we can, but I acknowledge the value you mention as well.

Frassle · 2023-10-31T08:31:50Z

Add support for a general Zip format for Pulumi templates in the CLI. This is an alternative to acquiring via a git ref.

This seems fine to me. Could then just allow any http://host/path.zip url.

but wanted to avoid building an API that would be difficult to iterate upon and test

I'm not sure I really follow why a zip file is any harder to test than the JSON files? A template is just a file tree, I don't think the CLI will ever accept anything more than that (given the constraints of what git gives us). A zip file seems a perfectly good representation of a file tree. Why does this make testing the AI service harder?

kpitzen · 2023-10-31T13:27:18Z

Add support for a general Zip format for Pulumi templates in the CLI. This is an alternative to acquiring via a git ref.

This seems fine to me. Could then just allow any http://host/path.zip url.

but wanted to avoid building an API that would be difficult to iterate upon and test

I'm not sure I really follow why a zip file is any harder to test than the JSON files? A template is just a file tree, I don't think the CLI will ever accept anything more than that (given the constraints of what git gives us). A zip file seems a perfectly good representation of a file tree. Why does this make testing the AI service harder?

It's not about the CLI when it comes to testing the service - if we're relying on the CLI for all of our iteration and testing, or worse, manually extracting an archive and checking the files each time, it's just going to be too slow. As I describe above, we can provide the capability to download a zipfile, but if we're already providing all of the required data in a way that provides some amount of structural guarantees, what's the downside? Do we have users asking to pulumi new zipfiles? I'd expect we'd have built that feature already if we had.

Anyway, I mentioned a possible middle-ground above. We can easily use an Accept header to branch off- if we're adamant that we want to use an opaque archive, we can certainly do so. We're biased towards shipping quickly per @AaronFriel 's mandate here, and this is what got us to where we are. If we want to do something different, we can, but it will slow us down.

sdk/go/common/workspace/templates_zip.go

…zipfiles We depend upon the user to provide a URL with a .zip suffix in order to determine which methodology to use, and assume the API serves this artifact on a PUT endpoint at the given path. We also provide a clean passthrough of any query parameters, though providing those on the command line can be a bit cumbersome.

lukehoban · 2023-10-31T16:47:20Z

I like the simplicity of the ZIP approach here.

Two things:

Can we add a description which captures what this change does?
Can we add tests?

kpitzen · 2023-10-31T16:58:29Z

I like the simplicity of the ZIP approach here.

Two things:
1. Can we add a description which captures what this change does?

2. Can we add tests?

Absolutely - just added a description above. Will add tests now.

Frassle · 2023-10-31T17:01:51Z

Probably reasonable to add the changelog entry for this now as well?

kpitzen · 2023-10-31T17:03:17Z

Probably reasonable to add the changelog entry for this now as well?

Since it's not tied to AI, I suppose we can. Do we want to consider the API contract (GET <url>.zip with the Accept header) stable?

Frassle · 2023-10-31T17:24:34Z

Do we want to consider the API contract (GET .zip with the Accept header) stable?

Yeh that seems reasonable to me.

kpitzen · 2023-10-31T18:55:09Z

@justinvp ++ @Frassle re-requested reviews w/ tests + changelog entry

AaronFriel

Just a question on the URL handling, the Zip test looks good though with the mock server.

sdk/go/common/workspace/templates_zip.go

kpitzen force-pushed the KP/AddPulumiNewAI branch 6 times, most recently from 61bb8fe to 5ce130d Compare October 30, 2023 20:04

kpitzen added the impact/no-changelog-required This issue doesn't require a CHANGELOG update label Oct 30, 2023

kpitzen force-pushed the KP/AddPulumiNewAI branch from 5ce130d to b5c6a9e Compare October 30, 2023 20:10

kpitzen marked this pull request as ready for review October 30, 2023 20:24

kpitzen requested review from justinvp, Frassle and AaronFriel October 30, 2023 20:24

kpitzen force-pushed the KP/AddPulumiNewAI branch from b5c6a9e to 2ea8a1f Compare October 30, 2023 20:58

Frassle reviewed Oct 30, 2023

View reviewed changes

sdk/go/common/workspace/templates_ai.go Outdated Show resolved Hide resolved

sdk/go/common/workspace/templates_ai.go Outdated Show resolved Hide resolved

kpitzen force-pushed the KP/AddPulumiNewAI branch from 2ea8a1f to f39222e Compare October 31, 2023 14:44

Frassle reviewed Oct 31, 2023

View reviewed changes

kpitzen force-pushed the KP/AddPulumiNewAI branch from f39222e to ac7884a Compare October 31, 2023 15:01

kpitzen changed the title ~~Adds ai:// scheme support to pulumi new~~ Adds remote .zip archive support to pulumi new Oct 31, 2023

kpitzen force-pushed the KP/AddPulumiNewAI branch 3 times, most recently from d95a734 to d2f39fa Compare October 31, 2023 15:31

kpitzen requested a review from Frassle October 31, 2023 15:36

Frassle reviewed Oct 31, 2023

View reviewed changes

sdk/go/common/workspace/templates_zip.go Outdated Show resolved Hide resolved

sdk/go/common/workspace/templates_zip.go Outdated Show resolved Hide resolved

kpitzen force-pushed the KP/AddPulumiNewAI branch from d2f39fa to f13801c Compare October 31, 2023 15:58

Frassle approved these changes Oct 31, 2023

View reviewed changes

kpitzen enabled auto-merge October 31, 2023 16:16

kpitzen added this pull request to the merge queue Oct 31, 2023

justinvp approved these changes Oct 31, 2023

View reviewed changes

AaronFriel removed this pull request from the merge queue due to a manual request Oct 31, 2023

Frassle removed the impact/no-changelog-required This issue doesn't require a CHANGELOG update label Oct 31, 2023

Adds tests for pulumi new zip

d1b020d

kpitzen requested review from justinvp and Frassle October 31, 2023 18:54

kpitzen force-pushed the KP/AddPulumiNewAI branch from a04b05a to e788c30 Compare October 31, 2023 19:00

AaronFriel reviewed Oct 31, 2023

View reviewed changes

sdk/go/common/workspace/templates_zip.go Outdated Show resolved Hide resolved

Adds changelog entry for zip support in pulumi new

fd084eb

kpitzen force-pushed the KP/AddPulumiNewAI branch from e788c30 to fd084eb Compare October 31, 2023 21:37

kpitzen enabled auto-merge October 31, 2023 21:41

kpitzen added this pull request to the merge queue Oct 31, 2023

Merged via the queue into master with commit f82b735 Oct 31, 2023
44 checks passed

kpitzen deleted the KP/AddPulumiNewAI branch October 31, 2023 22:45

justinvp mentioned this pull request Nov 2, 2023

Freeze v3.92.0 #14481

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds remote .zip archive support to `pulumi new` #14443

Adds remote .zip archive support to `pulumi new` #14443

kpitzen commented Oct 30, 2023 •

edited

pulumi-bot commented Oct 30, 2023 •

edited

kpitzen commented Oct 30, 2023

lukehoban commented Oct 31, 2023

kpitzen commented Oct 31, 2023

Frassle commented Oct 31, 2023

kpitzen commented Oct 31, 2023

lukehoban commented Oct 31, 2023

kpitzen commented Oct 31, 2023

Frassle commented Oct 31, 2023

kpitzen commented Oct 31, 2023

Frassle commented Oct 31, 2023

kpitzen commented Oct 31, 2023

AaronFriel left a comment

Adds remote .zip archive support to pulumi new #14443

Adds remote .zip archive support to pulumi new #14443

Conversation

kpitzen commented Oct 30, 2023 • edited

Description

Checklist

pulumi-bot commented Oct 30, 2023 • edited

Changelog

[uncommitted] (2023-10-31)

Features

kpitzen commented Oct 30, 2023

lukehoban commented Oct 31, 2023

kpitzen commented Oct 31, 2023

Frassle commented Oct 31, 2023

kpitzen commented Oct 31, 2023

lukehoban commented Oct 31, 2023

kpitzen commented Oct 31, 2023

Frassle commented Oct 31, 2023

kpitzen commented Oct 31, 2023

Frassle commented Oct 31, 2023

kpitzen commented Oct 31, 2023

AaronFriel left a comment

Choose a reason for hiding this comment

Adds remote .zip archive support to `pulumi new` #14443

Adds remote .zip archive support to `pulumi new` #14443

kpitzen commented Oct 30, 2023 •

edited

pulumi-bot commented Oct 30, 2023 •

edited