WIP: Readd project support to filestate #12134
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Changelog[uncommitted] (2023-03-15)Features
|
Frassle
force-pushed
the
fraser/filestate
branch
11 times, most recently
from
5bbacda
to
4b0bd82
Compare
Frassle
force-pushed
the
fraser/filestate
branch
5 times, most recently
from
e55c295
to
dbfd7cb
Compare
abhinav
force-pushed
the
fraser/filestate
branch
5 times, most recently
from
7825a05
to
ec0e2d7
Compare
abhinav
force-pushed
the
fraser/filestate
branch
from
ec0e2d7
to
b820b17
Compare
|
Current dependencies on/for this PR:
This comment was auto-generated by Graphite. |
abhinav
added a commit
that referenced
this pull request
Mar 13, 2023
Instead of using context.TODO everywhere, plumb context.Context into filestate.New and filestate.Login, and through that into other operations. Extracted from #12134 Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
force-pushed
the
fraser/filestate
branch
from
b820b17
to
1f3db66
Compare
bors bot
added a commit
that referenced
this pull request
Mar 22, 2023
12472: filestate: Track a state metadata file (.pulumi/Pulumi.yaml) r=abhinav a=abhinav We want the filestate backend to support project-scoped stacks, but we can't make the change as-is because it would break old states with new CLIs. To differentiate between old and new states, we've decided to introduce the concept of state metadata. This is a file under the path .pulumi/Pulumi.yaml that tracks metadata necessary for the filestate backend to operate. Initially, this contains just one field: `version`, with the initial value of 0 representing non-project or "legacy mode". This changes the filestate layout to track such a file, creating it if it doesn't exist with the default value of 0. In a future change, we'll introduce "version 1", which adds support for project-scoped stacks. If we ever need to make breaking changes to the layout, the version in this file will help the CLI decide whether it's allowed to handle that state bucket without corrupting it. Note that this differs slightly from the initial implementation of this functionality in #12134. Particularly, this idempotently ensures that a Pulumi.yaml exists, allowing `version: 0` to indicate legacy mode, versus the original implementation that treated absence of the file in a non-empty bucket as legacy mode. This drops the bucket.IsAccessible check from filestate.New because accessibility is now verified when we try to read the metadata file. Extracted from #12437 Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 22, 2023
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.
We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.
To accomplish this, and to plan for the future,
we introduce a 'Pulumi.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:
# .pulumi/Pulumi.yaml
version: 1
Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.
Notes:
- Stack references produced by filestate will shorten to
just the stack name if the project name for the stack
matches the currently selected project.
This required turning currentProject on localBackend
into an atomic pointer because otherwise
SetCurrentProject and localBackendReference.String may race.
Testing:
Besides included unit tests,
we duplicate some existsing tests that operate on fresh stores
to also run for legacy stores by emulating a pre-existing store.
Extracted from #12134
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 23, 2023
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.
We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.
To accomplish this, and to plan for the future,
we introduce a 'Pulumi.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:
# .pulumi/Pulumi.yaml
version: 1
Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.
Notes:
- Stack references produced by filestate will shorten to
just the stack name if the project name for the stack
matches the currently selected project.
This required turning currentProject on localBackend
into an atomic pointer because otherwise
SetCurrentProject and localBackendReference.String may race.
Extracted from #12134
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 23, 2023
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.
We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.
To accomplish this, and to plan for the future,
we introduce a 'Pulumi.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:
# .pulumi/Pulumi.yaml
version: 1
Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.
Notes:
- Stack references produced by filestate will shorten to
just the stack name if the project name for the stack
matches the currently selected project.
This required turning currentProject on localBackend
into an atomic pointer because otherwise
SetCurrentProject and localBackendReference.String may race.
Extracted from #12134
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 23, 2023
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.
We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.
To accomplish this, and to plan for the future,
we introduce a 'Pulumi.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:
# .pulumi/Pulumi.yaml
version: 1
Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.
Notes:
- Stack references produced by filestate will shorten to
just the stack name if the project name for the stack
matches the currently selected project.
This required turning currentProject on localBackend
into an atomic pointer because otherwise
SetCurrentProject and localBackendReference.String may race.
Extracted from #12134
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 23, 2023
Adds a new `pulumi state upgrade` command that will upgrade the state for the current state storage to the latest version supported by that format. This is a no-op for httpstate, but for filestate, this will migrate from unversioned storage to the project-based versioned format. The upgrade command is backed by a new Upgrade operation on the filestate Backend. The upgrade operation operates by effectively renaming stacks from legacy store to the new project store. UX notes: - We'll warn about stacks we failed to upgrade but keep going otherwise. - We'll print a loud warning before upgrading, informing users that the state will not be usable from older CLIs. Extracted from #12134 Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 24, 2023
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.
We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.
To accomplish this, and to plan for the future,
we introduce a 'Pulumi.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:
# .pulumi/Pulumi.yaml
version: 1
Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.
Notes:
- Stack references produced by filestate will shorten to
just the stack name if the project name for the stack
matches the currently selected project.
This required turning currentProject on localBackend
into an atomic pointer because otherwise
SetCurrentProject and localBackendReference.String may race.
Extracted from #12134
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 24, 2023
Adds a new `pulumi state upgrade` command that will upgrade the state for the current state storage to the latest version supported by that format. This is a no-op for httpstate, but for filestate, this will migrate from unversioned storage to the project-based versioned format. The upgrade command is backed by a new Upgrade operation on the filestate Backend. The upgrade operation operates by effectively renaming stacks from legacy store to the new project store. UX notes: - We'll warn about stacks we failed to upgrade but keep going otherwise. - We'll print a loud warning before upgrading, informing users that the state will not be usable from older CLIs. Extracted from #12134 Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 24, 2023
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.
We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.
To accomplish this, and to plan for the future,
we introduce a 'Pulumi.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:
# .pulumi/Pulumi.yaml
version: 1
Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.
Notes:
- Stack references produced by filestate will shorten to
just the stack name if the project name for the stack
matches the currently selected project.
This required turning currentProject on localBackend
into an atomic pointer because otherwise
SetCurrentProject and localBackendReference.String may race.
Extracted from #12134
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 24, 2023
Adds a new `pulumi state upgrade` command that will upgrade the state for the current state storage to the latest version supported by that format. This is a no-op for httpstate, but for filestate, this will migrate from unversioned storage to the project-based versioned format. The upgrade command is backed by a new Upgrade operation on the filestate Backend. The upgrade operation operates by effectively renaming stacks from legacy store to the new project store. UX notes: - We'll warn about stacks we failed to upgrade but keep going otherwise. - We'll print a loud warning before upgrading, informing users that the state will not be usable from older CLIs. Extracted from #12134 Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 28, 2023
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.
We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.
To accomplish this, and to plan for the future,
we introduce a 'Pulumi.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:
# .pulumi/Pulumi.yaml
version: 1
Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.
Notes:
- Stack references produced by filestate will shorten to
just the stack name if the project name for the stack
matches the currently selected project.
This required turning currentProject on localBackend
into an atomic pointer because otherwise
SetCurrentProject and localBackendReference.String may race.
Extracted from #12134
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 28, 2023
Adds a new `pulumi state upgrade` command that will upgrade the state for the current state storage to the latest version supported by that format. This is a no-op for httpstate, but for filestate, this will migrate from unversioned storage to the project-based versioned format. The upgrade command is backed by a new Upgrade operation on the filestate Backend. The upgrade operation operates by effectively renaming stacks from legacy store to the new project store. UX notes: - We'll warn about stacks we failed to upgrade but keep going otherwise. - We'll print a loud warning before upgrading, informing users that the state will not be usable from older CLIs. Extracted from #12134 Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 28, 2023
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.
We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.
To accomplish this, and to plan for the future,
we introduce a 'meta.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:
# .pulumi/meta.yaml
version: 1
Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.
Notes:
- Stack references produced by filestate will shorten to
just the stack name if the project name for the stack
matches the currently selected project.
This required turning currentProject on localBackend
into an atomic pointer because otherwise
SetCurrentProject and localBackendReference.String may race.
Extracted from #12134
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 28, 2023
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.
We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.
To accomplish this, and to plan for the future,
we introduce a 'meta.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:
# .pulumi/meta.yaml
version: 1
Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.
Notes:
- Stack references produced by filestate will shorten to
just the stack name if the project name for the stack
matches the currently selected project.
This required turning currentProject on localBackend
into an atomic pointer because otherwise
SetCurrentProject and localBackendReference.String may race.
Extracted from #12134
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 28, 2023
Adds a new `pulumi state upgrade` command that will upgrade the state for the current state storage to the latest version supported by that format. This is a no-op for httpstate, but for filestate, this will migrate from unversioned storage to the project-based versioned format. The upgrade command is backed by a new Upgrade operation on the filestate Backend. The upgrade operation operates by effectively renaming stacks from legacy store to the new project store. UX notes: - We'll warn about stacks we failed to upgrade but keep going otherwise. - We'll print a loud warning before upgrading, informing users that the state will not be usable from older CLIs. Extracted from #12134 Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 29, 2023
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.
We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.
To accomplish this, and to plan for the future,
we introduce a 'meta.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:
# .pulumi/meta.yaml
version: 1
Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.
Notes:
- Stack references produced by filestate will shorten to
just the stack name if the project name for the stack
matches the currently selected project.
This required turning currentProject on localBackend
into an atomic pointer because otherwise
SetCurrentProject and localBackendReference.String may race.
Extracted from #12134
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 29, 2023
Adds a new `pulumi state upgrade` command that will upgrade the state for the current state storage to the latest version supported by that format. This is a no-op for httpstate, but for filestate, this will migrate from unversioned storage to the project-based versioned format. The upgrade command is backed by a new Upgrade operation on the filestate Backend. The upgrade operation operates by effectively renaming stacks from legacy store to the new project store. UX notes: - We'll warn about stacks we failed to upgrade but keep going otherwise. - We'll print a loud warning before upgrading, informing users that the state will not be usable from older CLIs. Extracted from #12134 Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 29, 2023
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.
We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.
To accomplish this, and to plan for the future,
we introduce a 'meta.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:
# .pulumi/meta.yaml
version: 1
Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.
Notes:
- Stack references produced by filestate will shorten to
just the stack name if the project name for the stack
matches the currently selected project.
This required turning currentProject on localBackend
into an atomic pointer because otherwise
SetCurrentProject and localBackendReference.String may race.
Extracted from #12134
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 29, 2023
Adds a new `pulumi state upgrade` command that will upgrade the state for the current state storage to the latest version supported by that format. This is a no-op for httpstate, but for filestate, this will migrate from unversioned storage to the project-based versioned format. The upgrade command is backed by a new Upgrade operation on the filestate Backend. The upgrade operation operates by effectively renaming stacks from legacy store to the new project store. UX notes: - We'll warn about stacks we failed to upgrade but keep going otherwise. - We'll print a loud warning before upgrading, informing users that the state will not be usable from older CLIs. Extracted from #12134 Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 30, 2023
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.
We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.
To accomplish this, and to plan for the future,
we introduce a 'meta.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:
# .pulumi/meta.yaml
version: 1
Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.
Notes:
- Stack references produced by filestate will shorten to
just the stack name if the project name for the stack
matches the currently selected project.
This required turning currentProject on localBackend
into an atomic pointer because otherwise
SetCurrentProject and localBackendReference.String may race.
Extracted from #12134
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 30, 2023
Adds a new `pulumi state upgrade` command that will upgrade the state for the current state storage to the latest version supported by that format. This is a no-op for httpstate, but for filestate, this will migrate from unversioned storage to the project-based versioned format. The upgrade command is backed by a new Upgrade operation on the filestate Backend. The upgrade operation operates by effectively renaming stacks from legacy store to the new project store. UX notes: - We'll warn about stacks we failed to upgrade but keep going otherwise. - We'll print a loud warning before upgrading, informing users that the state will not be usable from older CLIs. Extracted from #12134 Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 31, 2023
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.
We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.
To accomplish this, and to plan for the future,
we introduce a 'meta.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:
# .pulumi/meta.yaml
version: 1
Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.
Notes:
- Stack references produced by filestate will shorten to
just the stack name if the project name for the stack
matches the currently selected project.
This required turning currentProject on localBackend
into an atomic pointer because otherwise
SetCurrentProject and localBackendReference.String may race.
Extracted from #12134
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Mar 31, 2023
Adds a new `pulumi state upgrade` command that will upgrade the state for the current state storage to the latest version supported by that format. This is a no-op for httpstate, but for filestate, this will migrate from unversioned storage to the project-based versioned format. The upgrade command is backed by a new Upgrade operation on the filestate Backend. The upgrade operation operates by effectively renaming stacks from legacy store to the new project store. UX notes: - We'll warn about stacks we failed to upgrade but keep going otherwise. - We'll print a loud warning before upgrading, informing users that the state will not be usable from older CLIs. Extracted from #12134 Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
bors bot
added a commit
that referenced
this pull request
Mar 31, 2023
12437: backend/filestate: Re-add project support r=abhinav a=abhinav
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.
We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.
To accomplish this, and to plan for the future,
we introduce a 'meta.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:
# .pulumi/meta.yaml
version: 1
Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.
Notes:
- Stack references produced by filestate will shorten to
just the stack name if the project name for the stack
matches the currently selected project.
Testing:
Besides included unit tests,
we duplicate some existing tests that operate on fresh stores
to also run for legacy stores by emulating a pre-existing store.
Environment variables:
This adds two new environment variables that affect behavior:
- PULUMI_SELF_MANAGED_STATE_NO_LEGACY_WARNING:
Suppresses the warning printed if a bucket contains both,
project-scoped and legacy stack files.
- PULUMI_SELF_MANAGED_STATE_LEGACY_LAYOUT:
Uses the legacy layout for new buckets even if they're empty
instead of project-scoped stacks.
Extracted from #12134
---
**Commits are split for reviewability.**
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
Co-authored-by: Fraser Waters <fraser@pulumi.com>
bors bot
added a commit
that referenced
this pull request
Mar 31, 2023
12437: backend/filestate: Re-add project support r=abhinav a=abhinav
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.
We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.
To accomplish this, and to plan for the future,
we introduce a 'meta.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:
# .pulumi/meta.yaml
version: 1
Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.
Notes:
- Stack references produced by filestate will shorten to
just the stack name if the project name for the stack
matches the currently selected project.
Testing:
Besides included unit tests,
we duplicate some existing tests that operate on fresh stores
to also run for legacy stores by emulating a pre-existing store.
Environment variables:
This adds two new environment variables that affect behavior:
- PULUMI_SELF_MANAGED_STATE_NO_LEGACY_WARNING:
Suppresses the warning printed if a bucket contains both,
project-scoped and legacy stack files.
- PULUMI_SELF_MANAGED_STATE_LEGACY_LAYOUT:
Uses the legacy layout for new buckets even if they're empty
instead of project-scoped stacks.
Extracted from #12134
---
**Commits are split for reviewability.**
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
Co-authored-by: Fraser Waters <fraser@pulumi.com>
bors bot
added a commit
that referenced
this pull request
Mar 31, 2023
12437: backend/filestate: Re-add project support r=abhinav a=abhinav
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.
We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.
To accomplish this, and to plan for the future,
we introduce a 'meta.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:
# .pulumi/meta.yaml
version: 1
Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.
Notes:
- Stack references produced by filestate will shorten to
just the stack name if the project name for the stack
matches the currently selected project.
Testing:
Besides included unit tests,
we duplicate some existing tests that operate on fresh stores
to also run for legacy stores by emulating a pre-existing store.
Environment variables:
This adds two new environment variables that affect behavior:
- PULUMI_SELF_MANAGED_STATE_NO_LEGACY_WARNING:
Suppresses the warning printed if a bucket contains both,
project-scoped and legacy stack files.
- PULUMI_SELF_MANAGED_STATE_LEGACY_LAYOUT:
Uses the legacy layout for new buckets even if they're empty
instead of project-scoped stacks.
Extracted from #12134
---
**Commits are split for reviewability.**
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
Co-authored-by: Fraser Waters <fraser@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Apr 1, 2023
Adds a new `pulumi state upgrade` command that will upgrade the state for the current state storage to the latest version supported by that format. This is a no-op for httpstate, but for filestate, this will migrate from unversioned storage to the project-based versioned format. The upgrade command is backed by a new Upgrade operation on the filestate Backend. The upgrade operation operates by effectively renaming stacks from legacy store to the new project store. UX notes: - We'll warn about stacks we failed to upgrade but keep going otherwise. - We'll print a loud warning before upgrading, informing users that the state will not be usable from older CLIs. Extracted from #12134 Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
bors bot
added a commit
that referenced
this pull request
Apr 1, 2023
12438: {cli, filestate}: Add 'state upgrade' command r=abhinav a=abhinav
Adds a new `pulumi state upgrade` command
that will upgrade the state for the current state storage
to the latest version supported by that format *concurrently*.
This is a no-op for httpstate,
but for filestate, this will migrate from unversioned storage
to the project-based versioned format.
It includes a confirmation message, informing users that
the state will be inaccessible from old CLIs after the upgrade.
They must type 'yes' to accept this.
<img width="428" alt="image" src="https://user-images.githubusercontent.com/41730/225780847-381bc9aa-a2a8-451e-bda1-2a942c4cb687.png">
A note on upgrade order:
We attempt to create the new meta.yaml file before moving the stack files.
This prevents us from leaving the bucket in an unreadable state
if we don't have write permission for the meta.yaml.
Extracted from #12134
Co-authored-by: Fraser Waters <fraser@pulumi.com>
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
bors bot
added a commit
that referenced
this pull request
Apr 1, 2023
12438: {cli, filestate}: Add 'state upgrade' command r=abhinav a=abhinav
Adds a new `pulumi state upgrade` command
that will upgrade the state for the current state storage
to the latest version supported by that format *concurrently*.
This is a no-op for httpstate,
but for filestate, this will migrate from unversioned storage
to the project-based versioned format.
It includes a confirmation message, informing users that
the state will be inaccessible from old CLIs after the upgrade.
They must type 'yes' to accept this.
<img width="428" alt="image" src="https://user-images.githubusercontent.com/41730/225780847-381bc9aa-a2a8-451e-bda1-2a942c4cb687.png">
A note on upgrade order:
We attempt to create the new meta.yaml file before moving the stack files.
This prevents us from leaving the bucket in an unreadable state
if we don't have write permission for the meta.yaml.
Extracted from #12134
Co-authored-by: Fraser Waters <fraser@pulumi.com>
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
abhinav
added a commit
that referenced
this pull request
Apr 1, 2023
{cli, filestate}: Add 'state upgrade' command
Adds a new `pulumi state upgrade` command
that will upgrade the state for the current state storage
to the latest version supported by that format *concurrently*.
This is a no-op for httpstate,
but for filestate, this will migrate from unversioned storage
to the project-based versioned format.
It includes a confirmation message, informing users that
the state will be inaccessible from old CLIs after the upgrade.
They must type 'yes' to accept this.
A note on upgrade order:
We attempt to create the new meta.yaml file before moving the stack files.
This prevents us from leaving the bucket in an unreadable state
if we don't have write permission for the meta.yaml.
Extracted from #12134
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds project support to the filestate backend.
For backwards compatibility, and to plan for the future,
this requires versioning the storage state.
We introduce a
.pulumi/Pulumi.yamlfile where we hold metadatalike the version of the filestate storage protocol.
Version 1 is the initial version that introduces project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.
Newly initialized states will use the new project-mode format.
Existing states will continue to run in the old, non-project mode.
State can be migrated to the new format with
pulumi state upgrade,at which point they will become incompatible with older CLIs.
For more graceful degradation, if an old CLI writes to an upgraded state
the CLI will warn about these files, and recommend re-running
upgrade.state upgradeshould accept an optional--projectoption for when a project for a stack cannot be determined automatically.state upgradeshould first warn the user that if they do this, older versions of the CLI will be unable to use the stack. Once they confirm with "y/n", only then should we upgrade the state.