Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for WAL archive with versions and persistant storage #63

Merged
merged 4 commits into from
Dec 2, 2019

Conversation

LamaAni
Copy link
Owner

@LamaAni LamaAni commented Nov 10, 2019

What:

Add support for WAL archiving of data-nodes.

Why

To allow datanodes to be stored to remote data, local data, or other storage, an automatic script for storing the WAL was added. Versions are supported to allow data to be versioned.

Missing: Auto-versioning of data on startup.

@LamaAni LamaAni requested a review from sstubbs November 10, 2019 21:45
@LamaAni
Copy link
Owner Author

LamaAni commented Nov 10, 2019

@sstubbs Some of these commits are just chaning /entry -> /scripts since it better describes that. But otherwise this adds functionality for safe storage, on the kubernetes side, where it would be a partial solution until we get everything else through. Feel free to comment on any unusable structure.

sstubbs
sstubbs previously approved these changes Nov 11, 2019
Copy link
Collaborator

@sstubbs sstubbs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK great. I'm sure you know this but just to double check taking backup via cp or rsync should have the server stopped.
https://www.postgresql.org/docs/10/backup-file.html The functionality I'm working on is using native postgres replication but this is only a slave copy for ha/distater recovery. Once what I'm busy with is working and we both happy with it, we could use this functionality you have added here to do backups at certain points rather than pg_dump.

@LamaAni
Copy link
Owner Author

LamaAni commented Nov 11, 2019

Actually, I am not aware of that. Why would cp or rsync stop the server?

As I understand, and in the settings posted the cp should happen on a partial file for Wal. If that is not the case then maybe this method should be avoided.

Can you give an example?

@sstubbs
Copy link
Collaborator

sstubbs commented Nov 11, 2019

Sorry I misread the commit. I thought you were copying the whole cluster as that was the only way in my mind to backup the gtm as I didn't realise it supported archive_command. Let me test this and see if it works. If the gtm supports archive_command then this is a brilliant solution and solves the backup issue.

@LamaAni LamaAni requested a review from sstubbs November 11, 2019 19:12
@LamaAni LamaAni dismissed sstubbs’s stale review November 11, 2019 19:13

Needs re-review see comment about purpose

@LamaAni
Copy link
Owner Author

LamaAni commented Nov 12, 2019

Needs re-review.

@LamaAni
Copy link
Owner Author

LamaAni commented Nov 22, 2019

@sstubbs Any update on this?

@LamaAni LamaAni merged commit a7c3118 into master Dec 2, 2019
@LamaAni LamaAni deleted the wal_archive branch December 2, 2019 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants