Feature: other S3 provider as state store then Amazon S3 #3592

CrystalMethod · 2019-11-30T07:14:58Z

We would like to use other S3 backend provider as state store then Amazon AWS S3, namely DigitalOcean an JFrog Artifactory, also Minio is a valid option. All attempts to adjust the S3 url property failed. It is always tried to connect to a host such as s3.amazon.com.

https://pulumi-community.slack.com/archives/C84L4E3N1/p1570861053113800

CrystalMethod · 2020-02-10T11:08:30Z

Any progress?

asyd · 2020-04-21T16:44:30Z

@CrystalMethod in case that help I successfully created a project in minio using:

/pulumi login 's3://minio/pulumi?region=fr&endpoint=domain.com'

Where minio url is minio.domain.com

bnason · 2020-04-22T13:54:07Z

@CrystalMethod in case that help I successfully created a project in minio using:
/pulumi login 's3://minio/pulumi?region=fr&endpoint=domain.com'
Where minio url is minio.domain.com

When trying to init a new stack using this method with Minio, I'm getting the following error

error: could not create stack: An IO error occurred during the current operation: blob (key ".pulumi/stacks/pulumi.json") (code=Unknown): InvalidAccessKeyId: The access key ID you provided does not exist in our records.
status code: 403, request id: 1608286DC236591B, host id:

How do I pass the Minio Access Key and Secret Key?
I tried to give the Minio bucket public read/write but that didn't fix it either

asyd · 2020-04-22T14:20:34Z

Edit ~/.aws/credentials to add a section, like:

[minio]
aws_access_key_id = xxx
aws_secret_access_key = xxx

Then, set environment variable AWS_PROFILE to minio before running pulumi

bnason · 2020-04-22T14:33:03Z

That did it, thanks!

CrystalMethod · 2020-04-22T17:30:47Z

Thanks @asyd. With your help I'm finally able to bootstrap a new project running pulumi new but pulumi up ends in an HTTP 404. I suspect it's because the Minio version is too old (2018-12-27T18:33:08Z). I will try a more recent version.

error: saving update info: blob (key ".pulumi/stacks/dev.json -> .pulumi/history/dev/dev-1587563593718859000.checkpoint.json") (code=Unknown): NoSuchBucket: The specified bucket does not exist
        status code: 404, request id: 160828AD664D26E3, host id:

CrystalMethod · 2020-04-29T16:47:32Z

No, even with latest Minio I get the error above when running pulumi up. Here is the stacktrace

error: saving update info: blob (key ".pulumi/stacks/dev.json -> .pulumi/history/dev/dev-1588177969928498000.checkpoint.json") (code=Unknown): NoSuchBucket: The specified bucket does not exist
        status code: 404, request id: 160A5772FB3BABB0, host id:
github.com/pulumi/pulumi/pkg/v2/backend/filestate.(*localBackend).apply
        /private/tmp/pulumi-20200416-24195-7o2f34/src/github.com/pulumi/pulumi/pkg/backend/filestate/backend.go:562
github.com/pulumi/pulumi/pkg/v2/backend.PreviewThenPromptThenExecute
        /private/tmp/pulumi-20200416-24195-7o2f34/src/github.com/pulumi/pulumi/pkg/backend/apply.go:211
github.com/pulumi/pulumi/pkg/v2/backend/filestate.(*localBackend).Update
        /private/tmp/pulumi-20200416-24195-7o2f34/src/github.com/pulumi/pulumi/pkg/backend/filestate/backend.go:420
github.com/pulumi/pulumi/pkg/v2/backend.UpdateStack
        /private/tmp/pulumi-20200416-24195-7o2f34/src/github.com/pulumi/pulumi/pkg/backend/stack.go:82
github.com/pulumi/pulumi/pkg/v2/backend/filestate.(*localStack).Update
        /private/tmp/pulumi-20200416-24195-7o2f34/src/github.com/pulumi/pulumi/pkg/backend/filestate/stack.go:72
main.newUpCmd.func1
        /private/tmp/pulumi-20200416-24195-7o2f34/src/github.com/pulumi/pulumi/pkg/cmd/pulumi/up.go:133
main.newUpCmd.func3
        /private/tmp/pulumi-20200416-24195-7o2f34/src/github.com/pulumi/pulumi/pkg/cmd/pulumi/up.go:370
github.com/pulumi/pulumi/sdk/v2/go/common/util/cmdutil.RunResultFunc.func1
        /private/tmp/pulumi-20200416-24195-7o2f34/src/github.com/pulumi/pulumi/sdk/go/common/util/cmdutil/exit.go:112
github.com/spf13/cobra.(*Command).execute
        /private/tmp/pulumi-20200416-24195-7o2f34/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:844
github.com/spf13/cobra.(*Command).ExecuteC
        /private/tmp/pulumi-20200416-24195-7o2f34/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:945
github.com/spf13/cobra.(*Command).Execute
        /private/tmp/pulumi-20200416-24195-7o2f34/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:885
main.main
        /private/tmp/pulumi-20200416-24195-7o2f34/src/github.com/pulumi/pulumi/pkg/cmd/pulumi/main.go:48
runtime.main
        /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/proc.go:203
runtime.goexit
        /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/asm_amd64.s:1373

CrystalMethod · 2020-04-29T17:08:15Z

Start latest Minio release using Docker:

$ docker run -p 9000:9000 minio/minio:RELEASE.2020-04-28T23-56-56Z server /data
...
Detected default credentials 'minioadmin:minioadmin'
...

Log into the Web UI and create a bucket pulumi.
Add default credentials to ~/.aws/credentials

...
[minio]
aws_access_key_id = minioadmin
aws_secret_access_key = minioadmin

Select the minio profile:

export AWS_PROFILE=minio

Add a subdomain to localhost in /etc/hosts

127.0.0.1        localhost minio.localhost

Login to Minio S3

$ pulumi login 's3://minio/pulumi?region=us-east-1&endpoint=http://localhost:9000'
Logged into Larss-MacBook-Pro.local as lars (s3://minio/pulumi?region=us-east-1&endpoint=http://localhost:9000)

Bootstrap a new Pulumi project

$ pulumi new kubernetes-typescript
...

Folders are created in Minio bucket:

$ pulumi up
...
error: saving update info: blob (key ".pulumi/stacks/dev.json -> .pulumi/history/dev/dev-1588177969928498000.checkpoint.json") (code=Unknown): NoSuchBucket: The specified bucket does not exist
        status code: 404, request id: 160A5772FB3BABB0, host id:

ydkn · 2020-05-13T16:10:01Z

Thank you @CrystalMethod - I got it working with OpenStack blob store this way except for one problem:

When saving the state at the end the following error is produced:

error: saving update info: blob (key ".pulumi/stacks/dev.json -> .pulumi/history/dev/dev-1589384917390913131.checkpoint.json") (code=Unknown): NoSuchBucket: status code: 404, request id: tx0000000000000006e1650-005ebc16d5-4a267cf-default, host id:

It seems the only the Copy operation is affected:

pulumi/pkg/backend/filestate/state.go

Line 415 in 9e91afc

return b.bucket.Copy(context.TODO(), checkpointFile, b.stackPath(name), nil)

other operations like the following are working and also the state itself is preserved and can be fetched from the backend again (e.g. after deleting ~/.pulumi locally):

pulumi/pkg/backend/filestate/state.go

Line 409 in 9e91afc

    
           if err = b.bucket.WriteAll(context.TODO(), historyFile, byts, nil); err != nil {

The corresponding history file (dev-1589384917390913131.history.json) therefore is also present on the s3 store.

Does somebody have an idea how to solve this?

kesavkolla · 2020-06-18T01:58:20Z

I'm getting the same error s3.CopyObject forbidden 403 with minio. I do have full rights on the bucket and still minio is throwing error of 403 forbidden. Has anyone figure out this?

stack72 · 2021-07-01T19:30:03Z

Hi Folks

Sorry this has been open so long - so we definitely have this working (especially for Minio)

https://www.pulumi.com/docs/intro/concepts/state/#logging-into-the-aws-s3-backend

As you can see from this, the url should be something similar to this:

pulumi login s3://<bucket-name>?endpoint=my.minio.local:8080&disableSSL=true&s3ForcePathStyle=true

Notice the s3ForcePathStyle=true as part of the url

But this works as expected

Please let us know if this is not the case and we can reopen and re-investigate

Paul

jaxxstorm · 2022-04-20T09:30:51Z

For those that end up on this issue because they're struggling with using a third party S3 compatible backend...

The AWS credentials you use for accessing the state are different from the credentials you use for provisioning infrastructure.

Credentials for provisioning infrastructure are set at the provider level, so you can define any of the options specified here:
https://www.pulumi.com/registry/packages/aws/api-docs/provider/

These credentials are NOT used for accessing the state backend.

So you can do the following:

define two aws profiles

aws_access_key_id = xxx
aws_secret_access_key = xxx
[aws]
aws_access_key_id = xxx
aws_secret_access_key = xxx```

And then do:

```pulumi config set aws:profile aws```

and then execute your program with the minio variables

```AWS_PROFILE=minio pulumi up```

SPFZ · 2023-11-16T13:58:49Z

For anyone having trouble with minio login like I did: make sure the aws_access_key_id does not contain underscores "_" (probably avoid special characters in general or use the minio auto-generated once).
I created a custom name for my access key with underscores in it "admin_access_key" and minio accepted it, however it did not work in pulumi and I got stuck for a while trying to figure out where I went wrong.

This worked for me:

$ mkdir ~/.aws
$ cat ~/.aws/credentials
[minio]
aws_access_key_id = g9YPeCpEdP5CqMdbgapS
aws_secret_access_key = pasXF5dW1EIvwEXXRlNgVd3EXSOVhgvEgDwm1JqX
$ export AWS_PROFILE="minio"
# bucket name was pulumi-test
$ pulumi login 's3://pulumi-test?endpoint=http://minio.myserverurl.com:9000&disableSSL=true&s3ForcePathStyle=true&region=de'
# pulumi created a folder .pulumi in the bucket with the file meta.yaml in it

trustos · 2024-05-21T20:57:19Z

For anyone trying to use Oracle S3 compatible buckets for login:

pulumi login s3://<bucket-name>?endpoint=https://<namespace>.compat.objectstorage.<oci-region>.oraclecloud.com&s3ForcePathStyle=true

The s3ForcePathStyle=true was something I missed before and It ended up throwing similar errors as others reported.
In my case I use it in Pulumi automation so I set the PULUMI_BACKEND_URL as the the s3 value.

CrystalMethod changed the title ~~Feature: other S3 provider then Amazon S3~~ Feature: other S3 provider as state store then Amazon S3 Dec 2, 2019

pgavlin self-assigned this Dec 2, 2019

leezen mentioned this issue May 11, 2020

Improve stability and functionality of self-managed state backends #4605

Open

19 tasks

lukehoban mentioned this issue Aug 11, 2020

Make it possible to use custom S3 endpoint url for cloud-backend #5155

Closed

stack72 assigned stack72 and unassigned pgavlin Jul 1, 2021

stack72 added kind/enhancement Improvements or new features resolution/fixed This issue was fixed labels Jul 1, 2021

stack72 added this to the 0.59 milestone Jul 1, 2021

stack72 closed this as completed Jul 1, 2021

jeremiepas mentioned this issue May 21, 2024

pulumi login with a custom s3 provider #16242

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: other S3 provider as state store then Amazon S3 #3592

Feature: other S3 provider as state store then Amazon S3 #3592

CrystalMethod commented Nov 30, 2019 •

edited

Loading

CrystalMethod commented Feb 10, 2020

asyd commented Apr 21, 2020

bnason commented Apr 22, 2020

asyd commented Apr 22, 2020

bnason commented Apr 22, 2020

CrystalMethod commented Apr 22, 2020 •

edited

Loading

CrystalMethod commented Apr 29, 2020

CrystalMethod commented Apr 29, 2020

ydkn commented May 13, 2020

kesavkolla commented Jun 18, 2020

stack72 commented Jul 1, 2021

jaxxstorm commented Apr 20, 2022

SPFZ commented Nov 16, 2023

trustos commented May 21, 2024 •

edited

Loading

Feature: other S3 provider as state store then Amazon S3 #3592

Feature: other S3 provider as state store then Amazon S3 #3592

Comments

CrystalMethod commented Nov 30, 2019 • edited Loading

CrystalMethod commented Feb 10, 2020

asyd commented Apr 21, 2020

bnason commented Apr 22, 2020

asyd commented Apr 22, 2020

bnason commented Apr 22, 2020

CrystalMethod commented Apr 22, 2020 • edited Loading

CrystalMethod commented Apr 29, 2020

CrystalMethod commented Apr 29, 2020

ydkn commented May 13, 2020

kesavkolla commented Jun 18, 2020

stack72 commented Jul 1, 2021

jaxxstorm commented Apr 20, 2022

SPFZ commented Nov 16, 2023

trustos commented May 21, 2024 • edited Loading

CrystalMethod commented Nov 30, 2019 •

edited

Loading

CrystalMethod commented Apr 22, 2020 •

edited

Loading

trustos commented May 21, 2024 •

edited

Loading