Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Readd project support to filestate #12134

Closed
wants to merge 16 commits into from
Closed

Commits on Mar 14, 2023

  1. filestate/internal: Use stack reference, not name

    filestate backend currently operates exclusively with stack names.
    All its internal pass around just the stack name, and nothing else.
    This makes it a bit difficult to add project support to the backend.
    
    This is a refactor in advance of adding project support,
    changing the internals of filestate to pass a stack reference around.
    It inspects the reference directly for all its operations.
    
    Note: This contains no behavioral changes.
    Name and FullyQualifiedName currently both return just the stack name.
    In a future change, once project name is incorporated into the object,
    FullyQualifiedName will be able to return `organization/$project/$name`.
    
    Extracted from #12134
    
    Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
    Frassle and abhinav committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    14fd8f9 View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2023

  1. backend/filestate: Re-add project support

    This re-adds project support back to the filestate backend.
    
    For backwards compatibility, and to plan for the future,
    this requires versioning the storage state.
    To do that, we introduce a .pulumi/Pulumi.yaml file
    where we hold metadata like the version of the filestate storage
    format.
    Version 1 is the initial version that introduces project support.
    
        # .pulumi/Pulumi.yaml
        version: 1
    
    If we ever need to make breaking changes to the storage protocol
    we can bump the format version.
    
    Newly initialized states will use the new project-mode format.
    Existing states will continue to run in the old, non-project mode.
    State can be migrated to the new format with `pulumi state upgrade`,
    at which point they will become incompatible with older CLIs.
    
    For more graceful degradation, if an old CLI writes to an upgraded state
    the CLI will warn about these files, and recommend re-running upgrade.
    
    Testing:
    Some tests inside filestate/backend were duplicated
    for legacy and project variants.
    Frassle authored and abhinav committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    67208a3 View commit details
    Browse the repository at this point in the history
  2. filestate/pulumiState: Move extraction into separate function

    Moves pulumiState extraction from a bucket
    into a separate independent ensurePulumiMeta function,
    reanming the type to pulumiMeta,
    and adds unit tests for success and failure cases of this function.
    abhinav committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    9b73977 View commit details
    Browse the repository at this point in the history
  3. test(filestate/New): warnings for legacy files in project mode

    Add a test that validates the warnings we print in project mode
    when there are legacy files in the bucket.
    abhinav committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    3f2d7bf View commit details
    Browse the repository at this point in the history
  4. cmd/state upgrade: Drop unused required argument

    The cobra command for 'state upgrade' specifies that
    it has one required positional argument.
    This isn't true--it takes no arguments, and it does nothing with 'args'.
    
    Switch to NoArgs and add a test.
    abhinav committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    98a0625 View commit details
    Browse the repository at this point in the history
  5. cmd/state upgrade: Refactor and add unit tests

    Refactors the 'state upgrade' command to use
    the pattern from #12400 and #11951
    and adds unit tests for its core functionality.
    abhinav committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    5663d93 View commit details
    Browse the repository at this point in the history
  6. refactor: Remove if-project-mode branches

    Instead of branching different code paths on
    whether we're in project mode or not,
    define an interface for those specific behaviors
    and extract code for those pieces into two separate implementations.
    
    This way, instead of:
    
        if projectMode { aProject() } else { aLegacy() }
        // ...
        if projectMode { bProject() } else { bLegacy() }
        // ...
        if projectMode { cProject() } else { cLegacy() }
    
    We do:
    
        var x X
        if projectMode { x = &projectX{} } else { x = &legacyX{} }
        x.a()
        x.b()
        x.c()
    abhinav committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    94d8686 View commit details
    Browse the repository at this point in the history
  7. Upgrade: Use existing logic to read old stacks

    Instead of duplciating the logic to read legacy stack files from the
    bucket, use the legacyReferenceStore to re-read the old references
    and upgrade them to project-level references.
    abhinav committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    29a9881 View commit details
    Browse the repository at this point in the history
  8. New: Re-use legacyReferenceStore to warn about old files

    Instead of looking for leftover legacy files in project mode manually,
    re-use the legacyReferenceStore logic to find them.
    abhinav committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    dbf2a69 View commit details
    Browse the repository at this point in the history
  9. localBackendReference: Don't track backend

    localBackendReference keeps track of the backend
    so that it can determine the current project
    and shorten the String representation of the reference.
    
    As far as I can tell,
    this doesn't need to be "current project at the time of calling String"
    and can be "current project when the reference was constructed."
    This switch allows us to turn localBackendReference
    back into a data-only object.
    abhinav committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    42f6a9e View commit details
    Browse the repository at this point in the history
  10. referenceStore: More control over layout, track on localBackendReference

    Let referenceStore decide paths of different files and directories
    rather than having the backend decide them
    based on whether localBackendReference.project is empty or not.
    
    To make renames from legacy to project-based stacks possible,
    we need localBackendReference to track the store that created it,
    and use that to determine old and new paths.
    abhinav committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    355ecf8 View commit details
    Browse the repository at this point in the history
  11. Revert "localBackendReference: Don't track backend"

    We actually need to track the localBackend
    because the current project for a localBackend
    does change after a reference is parsed.
    
    Includes a test for this change.
    abhinav committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    6271c74 View commit details
    Browse the repository at this point in the history
  12. Make .pulumi/{stacks,history,backup} into constants

    These paths are calculated on-demand repeatedly,
    but they never change.
    There's no reason for these to be bound to the backend implemntation.
    
    Turn them into top-level variables.
    (They're not constants because the output of filepath.Join cannot be a
    const.)
    abhinav committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    3a5ca77 View commit details
    Browse the repository at this point in the history
  13. currentProject: use an atomic pointer

    SetCurrentProject and localBackendReference.String have a data race
    in accessing the currentProject field.
    Turn it into an atomic.Pointer to address this data race.
    abhinav committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    c05fe58 View commit details
    Browse the repository at this point in the history
  14. projectReferenceStore: add ListProjects

    Adds a ListProjects methods to the projectReferenceStore
    which we re-use in DoesProjectExist and ListReferences
    rather than re-implementing that logic.
    abhinav committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    8e8b544 View commit details
    Browse the repository at this point in the history
  15. test: Move replicated legacy tests to separate file

    These don't need to pollute the primary file.
    Otherwise, future tests may feel compelled to duplicate for legacy and
    project variants.
    abhinav committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    5083e82 View commit details
    Browse the repository at this point in the history