Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: Prevent Leader from accidentally hanging up data inconsistency #597

Merged
merged 1 commit into from
Jul 14, 2022

Conversation

zhl003
Copy link
Collaborator

@zhl003 zhl003 commented Jul 14, 2022

What type of PR is this?

/enhancement

Which issue(s) this PR fixes?

Fixes #598

What this PR does?

Summary:

1. Add leader postStart,preStop cmd in xenon container   
2. Add leaderstop cmd in xenon conf
3. Adjust project structure

Special notes for your reviewer?

@zhl003 zhl003 added the enhancement New feature or request label Jul 14, 2022
@zhl003 zhl003 added this to the v2.2.1 milestone Jul 14, 2022
@zhl003 zhl003 self-assigned this Jul 14, 2022
@zhl003 zhl003 force-pushed the graceful_failover branch 5 times, most recently from c14eb2a to abf7e92 Compare July 14, 2022 05:08
   2. Add leaderstop cmd in xenon conf
   3. Adjust project structure
Copy link
Contributor

@acekingke acekingke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link

@qianfen2021 qianfen2021 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@acekingke acekingke merged commit 6076749 into radondb:main Jul 14, 2022
},
}

assert.Equal(t, lifeCycle, xenonCase.Lifecycle)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use "github.com/onsi/gomega" lib, Expect


"github.com/bitly/go-simplejson"
_ "github.com/go-sql-driver/mysql"
"github.com/jmoiron/sqlx"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_ "github.com/go-sql-driver/mysql" is ok, Do not use "github.com/jmoiron/sqlx"

cfg := loadConfiguration()
dsn := mysqlUser + ":" + cfg.password + "@tcp(" + mysqlHost + ")/"
server := MySQLServer{DSN: dsn}
Conn, err := server.GetNewDBConn()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

conn must close when finished

if err != nil {
log.Error(err)
}
IsLeader := IsLeader(Conn)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when error, to here is will crash.

}
// Step 1: disable event scheduler
log.Info("Disabling event scheduler")
stmt, err := SetEventScheduler(Conn, false)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the same question.

if err != nil {
log.Error(err)
}
log.Infof("set readonly to true: %s", stmt)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the same, when error , it will crash.

if err != nil {
log.Error(err)
}
log.Infof("%d,long running writes: %s", num, stmt)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the same, when error , it will crash.

leaderSet = gtid.ExecutedGTIDSet
}
}
db, err := server.GetNewDBConn()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

db need close.

if err != nil {
log.Fatal(err)
}
contents, err2 := ioutil.ReadAll(xenonConfFile)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

defer xenonConfFile.Close() should at here.

return false, fmt.Errorf("db connection failed after %s timeout", timeout)
case <-ticker.C:
_, err := sqlx.Connect("mysql", server.DSN)
if err == nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when success, need close.

@@ -8,4 +8,4 @@ spec:
# hostname if empty, use the leader as hostname
hostName: sample-mysql-0
clusterName: sample
# nfsServerAddress: ""
# nfsServerAddress: ""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

new lines.

@@ -37,4 +37,4 @@ EXPOSE 8801
VOLUME ["/var/lib/xenon", "/etc/xenon"]

ENTRYPOINT ["xenon"]
CMD ["-c", "/etc/xenon/xenon.json"]
CMD ["-c", "/etc/xenon/xenon.json"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

new lines.

for {
select {
case <-timeoutExceeded:
return false, fmt.Errorf("db connection failed after %s timeout", timeout)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

timeout is int64, use %s is very dangerous.

select {
case <-timeoutExceeded:
return false, fmt.Errorf("db connection failed after %s timeout", timeout)
case <-ticker.C:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need times to retry. 1 second is two frequent.

zhl003 pushed a commit to zhl003/radondb-mysql-kubernetes that referenced this pull request Aug 17, 2022
*: Prevent Leader from accidentally hanging up data inconsistency
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[bug] 🐛 force restart leader may cause data inconsistency
3 participants