Skip to content

Fix cloning from GCS #1176

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Nov 3, 2020
Merged

Fix cloning from GCS #1176

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
f842529
Merge pull request #1 from zalando/master
sagor999 Oct 10, 2020
351989a
Fix clone from gcs
sagor999 Oct 15, 2020
266c6e5
pass google credentials env var if using GS bucket
sagor999 Oct 15, 2020
ac4eb35
remove requirement for timezone as GCS returns timestamp in local tim…
sagor999 Oct 15, 2020
1390af0
Revert "remove requirement for timezone as GCS returns timestamp in l…
sagor999 Oct 15, 2020
7c50f15
update GCS documentation
sagor999 Oct 15, 2020
6742511
Merge branch 'improve_gcs_docs' into fix_clone_from_gcs
sagor999 Oct 19, 2020
1f7bcc7
remove sentence about logical backups
sagor999 Oct 27, 2020
a874151
reword pod environment configmap section
sagor999 Oct 29, 2020
79e73dd
fix documentation
sagor999 Oct 29, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions docs/administrator.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -685,6 +685,32 @@ aws_or_gcp:
...
```

### Setup pod environment configmap
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the documentation part needs to be formulated differently.

  • Title of the section should include "GCS" to indicate it's not just general configmap.

  • The operator doesn't have any preferences or suggestions about which method to use, so "we want to use WAL-G" need to be omitted. Instead it could be formulated as "To make operator work with GCS do the following: ...".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this paragraph is part of a bigger section about GCP so title is fine.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, then the title part is fine. But I would still ask to avoid indirect suggestions about using WAL-G (it could be a note though). Other than that looks good.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@erthalion I re-worded that section based on your feedback. I left a note as to why we are using WAL-G instead of WAL-E, so that end-user understands the reasoning, instead of just using "magical" configmap. Let me know if you agree with the changes.

Copy link
Contributor

@erthalion erthalion Oct 29, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sagor999 Yes, looks good, thanks. Do you have any references/docs we can also add to make the claim about WAL-G suited better for GCS look more respectable?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@erthalion valid point. I looked it up, and it seems like it is possible to make WAL-E work with GCS. So I have re-worded that phrase.


To make postgres-operator work with GCS, use following configmap:
```yml
apiVersion: v1
kind: ConfigMap
metadata:
name: pod-env-overrides
namespace: postgres-operator-system
data:
# Any env variable used by spilo can be added
USE_WALG_BACKUP: "true"
USE_WALG_RESTORE: "true"
CLONE_USE_WALG_RESTORE: "true"
```
This configmap will instruct operator to use WAL-G, instead of WAL-E, for backup and restore.

Then provide this configmap in postgres-operator settings:
```yml
...
# namespaced name of the ConfigMap with environment variables to populate on every pod
pod_environment_configmap: "postgres-operator-system/pod-env-overrides"
...
```


## Sidecars for Postgres clusters

A list of sidecars is added to each cluster created by the operator. The default
Expand Down
28 changes: 24 additions & 4 deletions pkg/cluster/k8sres.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1752,11 +1752,31 @@ func (c *Cluster) generateCloneEnvironment(description *acidv1.CloneDescription)
msg := "Figure out which S3 bucket to use from env"
c.logger.Info(msg, description.S3WalPath)

if c.OpConfig.WALES3Bucket != "" {
envs := []v1.EnvVar{
{
Name: "CLONE_WAL_S3_BUCKET",
Value: c.OpConfig.WALES3Bucket,
},
}
result = append(result, envs...)
} else if c.OpConfig.WALGSBucket != "" {
envs := []v1.EnvVar{
{
Name: "CLONE_WAL_GS_BUCKET",
Value: c.OpConfig.WALGSBucket,
},
{
Name: "CLONE_GOOGLE_APPLICATION_CREDENTIALS",
Value: c.OpConfig.GCPCredentials,
},
}
result = append(result, envs...)
} else {
c.logger.Error("Cannot figure out S3 or GS bucket. Both are empty.")
}

envs := []v1.EnvVar{
{
Name: "CLONE_WAL_S3_BUCKET",
Value: c.OpConfig.WALES3Bucket,
},
{
Name: "CLONE_WAL_BUCKET_SCOPE_SUFFIX",
Value: getBucketScopeSuffix(description.UID),
Expand Down