Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add daemon mode for wal-push command #1353

Merged
merged 23 commits into from Oct 28, 2022

Conversation

rogaliiik
Copy link
Contributor

Database name

postgresql15

Pull request description

Added daemon command for archiving wal segments

@rogaliiik rogaliiik requested a review from a team as a code owner October 7, 2022 10:51
cmd/pg/daemon.go Show resolved Hide resolved
cmd/pg/daemon.go Outdated Show resolved Hide resolved
cmd/pg/daemon.go Outdated Show resolved Hide resolved
cmd/pg/daemon.go Outdated Show resolved Hide resolved
docker/pg_tests/scripts/tests/daemon_test.sh Show resolved Hide resolved
internal/databases/postgres/daemon_handler.go Outdated Show resolved Hide resolved
internal/databases/postgres/daemon_handler.go Outdated Show resolved Hide resolved
cmd/pg/daemon.go Outdated Show resolved Hide resolved
internal/databases/postgres/daemon_handler.go Outdated Show resolved Hide resolved
internal/databases/postgres/daemon_handler.go Outdated Show resolved Hide resolved
internal/databases/postgres/daemon_handler.go Outdated Show resolved Hide resolved
internal/databases/postgres/daemon_handler.go Outdated Show resolved Hide resolved
@rogaliiik rogaliiik force-pushed the daemon-master-dev branch 4 times, most recently from a9f7e95 to 66c110d Compare October 21, 2022 16:09
@rogaliiik rogaliiik force-pushed the daemon-master-dev branch 14 times, most recently from af155a8 to 912cf6a Compare October 24, 2022 11:43
else
echo "Error in WAL-G response"
exit 1
fi
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also need to check that WAL has been archived correctly.

cmd/pg/wal_push.go Outdated Show resolved Hide resolved

// daemonCmd represents the daemon archive command
var daemonCmd = &cobra.Command{
Use: "daemon daemon_socket_path",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add some information about this command to the documentation.

internal/databases/postgres/daemon_handler.go Show resolved Hide resolved
func (h *ArchiveMessageHandler) Handle(messageBody []byte) error {
err := messageValidation(messageBody)
if err != nil {
tracelog.ErrorLogger.Printf("Incorrect message: %v\n", err)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can just log the error in the upper level.
Here you can use the fmt.Errorf("msg: %w", err) to wrap error with some message.

For example:

if err != nil {
    return fmt.Errorf("message validation failed: %w", err)
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same issue in other places below.

tracelog.ErrorLogger.Fatal("Error on listening socket:", err)
}
for {
fd, err := l.Accept()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we would want to terminate the running daemon?

I suggest to add the signal listener and listend for SIGTERM to perform a graceful shutdown. But looks like it is a feature for the next PR, not for the MVP.

internal/databases/postgres/daemon_handler.go Outdated Show resolved Hide resolved
messageType, messageBody, err := messageReader.Next()
if err != nil {
tracelog.ErrorLogger.Printf("Failed to read message from client %s, err: %v\n", c.RemoteAddr(), err)
_, _ = c.Write([]byte{'E'})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do not ignore the possible errors

messageHandler := GetMessageHandler(messageType, c, uploader)
if messageHandler == nil {
tracelog.ErrorLogger.Printf("Unexpected message type: %s", string(messageType))
_, _ = c.Write([]byte{'E'})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Create these mesage types as well:

type ErrorType SocketMessageType = 'E'
type ... SocketMessageType = 'F'
type ... SocketMessageType = 'O'
...

internal/databases/postgres/daemon_handler.go Outdated Show resolved Hide resolved
}

func messageValidation(messageBody []byte) error {
if len(messageBody) < 24 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, we archive not only WALs but other files with length other than 24.

I think we can remove this validation.

@usernamedt usernamedt changed the title Daemon master dev code review Add daemon mode for wal-push command Oct 28, 2022
@usernamedt usernamedt merged commit 925d2c4 into wal-g:master Oct 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants