backend/filestate: Re-add project support #12437

abhinav · 2023-03-15T00:56:27Z

This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.

We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.

To accomplish this, and to plan for the future,
we introduce a 'meta.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:

# .pulumi/meta.yaml
version: 1

Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.

Notes:

Stack references produced by filestate will shorten to
just the stack name if the project name for the stack
matches the currently selected project.

Testing:
Besides included unit tests,
we duplicate some existing tests that operate on fresh stores
to also run for legacy stores by emulating a pre-existing store.

Environment variables:
This adds two new environment variables that affect behavior:

PULUMI_SELF_MANAGED_STATE_NO_LEGACY_WARNING:
Suppresses the warning printed if a bucket contains both,
project-scoped and legacy stack files.
PULUMI_SELF_MANAGED_STATE_LEGACY_LAYOUT:
Uses the legacy layout for new buckets even if they're empty
instead of project-scoped stacks.

Extracted from #12134

Commits are split for reviewability.

abhinav · 2023-03-15T00:56:44Z

Current dependencies on/for this PR:

master
- PR filestate: Don't write metadata for legacy layouts #12537
  - PR filestate: Rename Pulumi.yaml to meta.yaml #12538
    - PR backend/filestate: Re-add project support #12437 👈
      - PR {cli, filestate}: Add 'state upgrade' command #12438
        
        PR refactor(filestate): Don't use t.Setenv in tests #12497
        
        PR filestate/projectStore: Optimize ListReferences #12496

This comment was auto-generated by Graphite.

pulumi-bot · 2023-03-16T21:46:25Z

Changelog

[uncommitted] (2023-03-31)

Features

[backend/filestate] Add support for project-scoped stacks.
Newly initialized storage will automatically use this mode.
Set PULUMI_SELF_MANAGED_STATE_LEGACY_LAYOUT=1 to opt-out of this.
This mode needs write access to the root of the .pulumi directory;
if you're using a cloud storage, be sure to update your ACLs.

#12437

Miscellaneous

[backend/filestate] Print a warning if a project-scoped storage has non-project stacks in it.
Disable this warning by setting PULUMI_SELF_MANAGED_STATE_NO_LEGACY_WARNING=1.

#12437

abhinav · 2023-03-22T15:01:54Z

Switching back to draft to make some changes.

We want the filestate backend to support project-scoped stacks, but we can't make the change as-is because it would break old states with new CLIs. To differentiate between old and new states, we've decided to introduce the concept of state metadata. This is a file under the path .pulumi/Pulumi.yaml that tracks metadata necessary for the filestate backend to operate. Initially, this contains just one field: `version`, with the initial value of 0 representing non-project or "legacy mode". This changes the filestate layout to track such a file, creating it if it doesn't exist with the default value of 0. In a future change, we'll introduce "version 1", which adds support for project-scoped stacks. If we ever need to make breaking changes to the layout, the version in this file will help the CLI decide whether it's allowed to handle that state bucket without corrupting it. Note that this differs slightly from the initial implementation of this functionality in #12134. Particularly, this idempotently ensures that a Pulumi.yaml exists, allowing `version: 0` to indicate legacy mode, versus the original implementation that treated absence of the file in a non-empty bucket as legacy mode. This drops the bucket.IsAccessible check from filestate.New because accessibility is now verified when we try to read the metadata file. Extracted from #12437

Adds a new constructor for filestate backend that accepts optional arguments in an Options struct. The only optional argument right now is the current project for the backend. This is a prefactor in anticipation of #12437 where we add another option -- primarily in tests. Extracted from #12437

Frassle · 2023-03-29T09:12:12Z

...g/20230128--backend-filestate--the-filestate-backend-now-supports-project-scoped-stacks.yaml

+- type: feat
+  scope: backend/filestate
+  description: |
+    Add support for project-scoped stacks.


We should add a comment here that write access to the root of the bucket is now needed to write "meta.yaml"

Frassle · 2023-03-29T09:13:11Z

pkg/backend/filestate/backend.go

@@ -65,6 +66,28 @@ import (
 // to enable gzip compression when using the filestate backend.
 const PulumiFilestateGzipEnvVar = "PULUMI_SELF_MANAGED_STATE_GZIP"

+// This section contains names of environment variables


Suggested change

// This section contains names of environment variables

// TODO[pulumi/issues/12539]: This section contains names of environment variables

Frassle · 2023-03-29T09:13:35Z

pkg/backend/filestate/backend.go

+//
+// These must all be registered in common/env so that they're available
+// with the 'pulumi env' command.
+// However, we don't currently use env.Value() to access their values


Yeh we should overhaul the idea of "env" so we can inject it

Created #12548 to track

pkg/backend/filestate/backend.go

pkg/backend/filestate/meta.go

The filestate backend now defaults to using project mode for new stores. This has a side effect of migrating stores in all existing tests to project mode. As a result of this, we lost some of the legacy test. To get around this, create legacy copies of affected tests. We run them in legacy mode by ensuring that the bucket is non-empty with a marker file.

This re-adds project support back to the filestate backend by implementing a new referenceStore: projectReferenceStore. We will use this reference store for all new filestate stores. Existing states will continue to use the legacyReferenceStore. To accomplish this, and to plan for the future, we introduce a 'meta.yaml' file inside the .pulumi directory. This file contains metadata about the storage state. Currently, this only holds a version number: # .pulumi/meta.yaml version: 1 Version 1 is the number we've chosen for the initial release of project support. If we ever need to make breaking changes to the storage protocol we can bump the format version. Notes: - Stack references produced by filestate will shorten to just the stack name if the project name for the stack matches the currently selected project. This required turning currentProject on localBackend into an atomic pointer because otherwise SetCurrentProject and localBackendReference.String may race. Extracted from #12134 Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>

It's possible to end up in a mixed state where some stacks are project-scoped and others aren't. For instance, imagine the scenario: - *User A* sets up a new state with the latest version of the CLI and brings up a new stack - *User B* brings up a stack in that state using an older version; the older version doesn't know about the .pulumi/meta.yaml so it doesn't explode and writes a legacy stack file At this point, the state is in mixed state. To help users detect (and eventually resolve) this issue, we'll print a warning when the backend is initialized. It'll find and warn users about any legacy files. They can opt out of this check by setting an environment variable. This will help transition users to project-scoped stacks until we're ready to delete this environment variable and the check.

With project support added, filestate will default to project-scoped stacks for all newly initialized buckets. This is desirable long-term, but for the initial release, we'd like users to have an escape hatch to go back to the old layout until they've had a change to migrate. This adds the ability for users to opt-out of this feature by setting an environment variable. Note that this only applies to new buckets. You cannot opt out of this feature for a bucket that is already using project-scoped stacks.

abhinav · 2023-03-31T21:03:16Z

bors merge

12437: backend/filestate: Re-add project support r=abhinav a=abhinav This re-adds project support back to the filestate backend by implementing a new referenceStore: projectReferenceStore. We will use this reference store for all new filestate stores. Existing states will continue to use the legacyReferenceStore. To accomplish this, and to plan for the future, we introduce a 'meta.yaml' file inside the .pulumi directory. This file contains metadata about the storage state. Currently, this only holds a version number: # .pulumi/meta.yaml version: 1 Version 1 is the number we've chosen for the initial release of project support. If we ever need to make breaking changes to the storage protocol we can bump the format version. Notes: - Stack references produced by filestate will shorten to just the stack name if the project name for the stack matches the currently selected project. Testing: Besides included unit tests, we duplicate some existing tests that operate on fresh stores to also run for legacy stores by emulating a pre-existing store. Environment variables: This adds two new environment variables that affect behavior: - PULUMI_SELF_MANAGED_STATE_NO_LEGACY_WARNING: Suppresses the warning printed if a bucket contains both, project-scoped and legacy stack files. - PULUMI_SELF_MANAGED_STATE_LEGACY_LAYOUT: Uses the legacy layout for new buckets even if they're empty instead of project-scoped stacks. Extracted from #12134 --- **Commits are split for reviewability.** Co-authored-by: Abhinav Gupta <abhinav@pulumi.com> Co-authored-by: Fraser Waters <fraser@pulumi.com>

bors · 2023-03-31T21:34:29Z

Build failed:

bors-ok

abhinav · 2023-03-31T21:38:05Z

    integration_test.go:139: STDERR: error: could not create stack: the stack is currently locked by 1 lock(s). Either wait for the other process(es) to end or delete the lock file with `pulumi cancel`.
          file:///C:/Users/RUNNER~1/AppData/Local/Temp/test-env2095457129/.pulumi/locks/organization/testing-config/testing/d327eb28-d0bb-4af1-a629-cb15113350ab.json: created by runner\runneradmin@runner (pid 6032) at 2023-03-31T21:22:26Z
    integration_test.go:139: Ran command pulumi args [stack init testing] and expected success. Instead got failure.

This looks like a transient failure.

abhinav · 2023-03-31T21:38:26Z

bors retry

12437: backend/filestate: Re-add project support r=abhinav a=abhinav This re-adds project support back to the filestate backend by implementing a new referenceStore: projectReferenceStore. We will use this reference store for all new filestate stores. Existing states will continue to use the legacyReferenceStore. To accomplish this, and to plan for the future, we introduce a 'meta.yaml' file inside the .pulumi directory. This file contains metadata about the storage state. Currently, this only holds a version number: # .pulumi/meta.yaml version: 1 Version 1 is the number we've chosen for the initial release of project support. If we ever need to make breaking changes to the storage protocol we can bump the format version. Notes: - Stack references produced by filestate will shorten to just the stack name if the project name for the stack matches the currently selected project. Testing: Besides included unit tests, we duplicate some existing tests that operate on fresh stores to also run for legacy stores by emulating a pre-existing store. Environment variables: This adds two new environment variables that affect behavior: - PULUMI_SELF_MANAGED_STATE_NO_LEGACY_WARNING: Suppresses the warning printed if a bucket contains both, project-scoped and legacy stack files. - PULUMI_SELF_MANAGED_STATE_LEGACY_LAYOUT: Uses the legacy layout for new buckets even if they're empty instead of project-scoped stacks. Extracted from #12134 --- **Commits are split for reviewability.** Co-authored-by: Abhinav Gupta <abhinav@pulumi.com> Co-authored-by: Fraser Waters <fraser@pulumi.com>

abhinav · 2023-03-31T21:58:10Z

bors cancel

bors · 2023-03-31T21:58:11Z

Canceled.

abhinav · 2023-03-31T21:59:20Z

Failed again. Digging.

=== FAIL: integration TestConfigSave (0.45s)
    environment.go:96: Created new test environment:  C:\Users\RUNNER~1\AppData\Local\Temp\test-env750429982
    integration_acceptance_test.go:52: Running command pulumi login --cloud-url file://C:/Users/RUNNER~1/AppData/Local/Temp/test-env750429982
    integration_acceptance_test.go:53: Running command pulumi stack init testing-2
    integration_acceptance_test.go:53: Run Error: exit status 0xffffffff
    integration_acceptance_test.go:53: STDOUT: 
    integration_acceptance_test.go:53: STDERR: error: could not create stack: the stack is currently locked by 1 lock(s). Either wait for the other process(es) to end or delete the lock file with `pulumi cancel`.
          file:///C:/Users/RUNNER~1/AppData/Local/Temp/test-env750429982/.pulumi/locks/organization/testing-config/testing-2/4178622a-c06f-40f6-88b4-2ae3217b5ca4.json: created by runner\runneradmin@runner (pid 2856) at 2023-03-31T21:52:52Z
    integration_acceptance_test.go:53: Ran command pulumi args [stack init testing-2] and expected success. Instead got failure.
=== FAIL: integration TestExcludeProtected (0.45s)
    environment.go:96: Created new test environment:  C:\Users\RUNNER~1\AppData\Local\Temp\test-env2045550713
    integration_test.go:627: Running command pulumi login --cloud-url file://C:/Users/RUNNER~1/AppData/Local/Temp/test-env2045550713
    integration_test.go:629: Running command pulumi stack init dev
    integration_test.go:629: Run Error: exit status 0xffffffff
    integration_test.go:629: STDOUT: 
    integration_test.go:629: STDERR: error: could not create stack: the stack is currently locked by 1 lock(s). Either wait for the other process(es) to end or delete the lock file with `pulumi cancel`.
          file:///C:/Users/RUNNER~1/AppData/Local/Temp/test-env2045550713/.pulumi/locks/organization/exclude-protected/dev/ca3097c7-262b-48e5-893e-465c5ffb2f51.json: created by runner\runneradmin@runner (pid [70](https://github.com/pulumi/pulumi/actions/runs/4579375065/jobs/8087120837#step:32:70)48) at 2023-03-31T21:53:18Z
    integration_test.go:629: Ran command pulumi args [stack init dev] and expected success. Instead got failure.

Frassle · 2023-03-31T22:16:19Z

could not create stack: the stack is currently locked by 1 lock(s)

Are the lock files project scoped?

Integration tests for filestate fail on Windows because the stack reference path is a mix of `/` and `\\` symbols. e.g. ``` want: .pulumi/locks/organization/testing-config/testing/640b9586-d794-4cce-a586-9ab58549d3d3.json got: .pulumi/locks/organization\testing-config\testing/640b9586-d794-4cce-a586-9ab58549d3d3.json ``` On Windows, the `<org>/<proj>/<name>` combo gets turned to `<org>\<proj>\<name>` by `stackLockDir(stackName)`. This is normally not a problem because the result is fed into the blob.Bucket, which handles both, but here we do exact string matching where this distinction matters. To fix, make sure to convert the path to `/` form before comparison.

abhinav · 2023-03-31T23:40:34Z

Yes, the lock files are project scoped. They use fully qualified paths for the lock path.
The issue was more fun than that:

want: .pulumi/locks/organization/testing-config/testing/640b9586-d794-4cce-a586-9ab58549d3d3.json
 got: .pulumi/locks/organization\testing-config\testing/640b9586-d794-4cce-a586-9ab58549d3d3.json

stackLockDir always flips to OS filepaths for the $org/$proj/$name.
I've added a fix for this.

abhinav · 2023-03-31T23:53:24Z

bors merge

bors · 2023-04-01T00:28:28Z

Build succeeded:

bors-ok

Adds a new blog post intended to be published this Wednesday (3/29) on the same day the feature will be released. Part of pulumi/pulumi#12173 Related Issues/PRs: - pulumi/pulumi#2522 - pulumi/pulumi#12437 - pulumi/pulumi#12438 Co-authored-by: Abhinav Gupta <abhinav@pulumi.com> Co-authored-by: George Huang <george@pulumi.com> Co-authored-by: Luke Hoban <luke@pulumi.com>

abhinav mentioned this pull request Mar 15, 2023

{cli, filestate}: Add 'state upgrade' command #12438

Merged

abhinav force-pushed the abhinav/filestate-legacy-store branch from 3a44ed4 to c8e2e2c Compare March 16, 2023 21:45

abhinav force-pushed the abhinav/filestate-project-support branch from d0ac7aa to 90e2493 Compare March 16, 2023 21:45

abhinav force-pushed the abhinav/filestate-legacy-store branch from c8e2e2c to 2ecb991 Compare March 16, 2023 22:53

abhinav force-pushed the abhinav/filestate-project-support branch from 90e2493 to d2ff37e Compare March 16, 2023 23:34

abhinav marked this pull request as ready for review March 16, 2023 23:35

abhinav requested a review from a team March 16, 2023 23:38

abhinav force-pushed the abhinav/filestate-legacy-store branch from 2ecb991 to 91168d9 Compare March 17, 2023 15:59

abhinav force-pushed the abhinav/filestate-project-support branch from d2ff37e to 0b0ca33 Compare March 17, 2023 15:59

abhinav force-pushed the abhinav/filestate-legacy-store branch from 91168d9 to 1314f3c Compare March 17, 2023 16:24

abhinav force-pushed the abhinav/filestate-project-support branch from 0b0ca33 to e6140fc Compare March 17, 2023 16:24

abhinav force-pushed the abhinav/filestate-legacy-store branch from 1314f3c to 0417219 Compare March 17, 2023 19:05

Base automatically changed from abhinav/filestate-legacy-store to master March 17, 2023 20:24

abhinav force-pushed the abhinav/filestate-project-support branch 2 times, most recently from 6aeb938 to 76497ac Compare March 21, 2023 21:29

abhinav marked this pull request as draft March 22, 2023 15:01

abhinav force-pushed the abhinav/filestate-project-support branch from 76497ac to 1398e85 Compare March 22, 2023 18:55

abhinav mentioned this pull request Mar 22, 2023

filestate: Track a state metadata file (.pulumi/Pulumi.yaml) #12472

Merged

This was referenced Mar 22, 2023

filestate: Add NewWithOptions constructor #12473

Closed

filestate: Make currentProject an atomic.Pointer #12474

Merged

abhinav force-pushed the abhinav/filestate-project-support branch from bfa7954 to 83fa36c Compare March 29, 2023 01:03

Base automatically changed from abhinav/filestate-pulumi-yaml-to-meta to master March 29, 2023 01:22

Frassle reviewed Mar 29, 2023

View reviewed changes

abhinav mentioned this pull request Mar 29, 2023

perf(filestate): DoesProjectExist should not list all #12547

Closed

abhinav force-pushed the abhinav/filestate-project-support branch 2 times, most recently from 6d584f1 to 55449d2 Compare March 30, 2023 20:04

abhinav and others added 4 commits March 31, 2023 13:21

abhinav force-pushed the abhinav/filestate-project-support branch from 55449d2 to 20e8917 Compare March 31, 2023 20:22

justinvp approved these changes Mar 31, 2023

View reviewed changes

bors bot merged commit fa8fee4 into master Apr 1, 2023

bors bot deleted the abhinav/filestate-project-support branch April 1, 2023 00:28

justinvp mentioned this pull request Apr 10, 2023

Support a pseudo-org for StackReference and GetOrganization API on filestate #11390

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

backend/filestate: Re-add project support #12437

backend/filestate: Re-add project support #12437

abhinav commented Mar 15, 2023 •

edited

Loading

abhinav commented Mar 15, 2023 •

edited

Loading

pulumi-bot commented Mar 16, 2023 •

edited

Loading

abhinav commented Mar 22, 2023

Frassle Mar 29, 2023

Frassle Mar 29, 2023

Frassle Mar 29, 2023

abhinav Mar 29, 2023

abhinav commented Mar 31, 2023

bors bot commented Mar 31, 2023

abhinav commented Mar 31, 2023

abhinav commented Mar 31, 2023

abhinav commented Mar 31, 2023

bors bot commented Mar 31, 2023

abhinav commented Mar 31, 2023

Frassle commented Mar 31, 2023

abhinav commented Mar 31, 2023

abhinav commented Mar 31, 2023

bors bot commented Apr 1, 2023

	// This section contains names of environment variables
	// TODO[pulumi/issues/12539]: This section contains names of environment variables

backend/filestate: Re-add project support #12437

backend/filestate: Re-add project support #12437

Conversation

abhinav commented Mar 15, 2023 • edited Loading

abhinav commented Mar 15, 2023 • edited Loading

pulumi-bot commented Mar 16, 2023 • edited Loading

Changelog

[uncommitted] (2023-03-31)

Features

Miscellaneous

abhinav commented Mar 22, 2023

Frassle Mar 29, 2023

Choose a reason for hiding this comment

Frassle Mar 29, 2023

Choose a reason for hiding this comment

Frassle Mar 29, 2023

Choose a reason for hiding this comment

abhinav Mar 29, 2023

Choose a reason for hiding this comment

abhinav commented Mar 31, 2023

bors bot commented Mar 31, 2023

abhinav commented Mar 31, 2023

abhinav commented Mar 31, 2023

abhinav commented Mar 31, 2023

bors bot commented Mar 31, 2023

abhinav commented Mar 31, 2023

Frassle commented Mar 31, 2023

abhinav commented Mar 31, 2023

abhinav commented Mar 31, 2023

bors bot commented Apr 1, 2023

abhinav commented Mar 15, 2023 •

edited

Loading

abhinav commented Mar 15, 2023 •

edited

Loading

pulumi-bot commented Mar 16, 2023 •

edited

Loading