Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZOOKEEPER-2824: `FileChannel#size` info should be added to `FileTxnLog#commit` to solve the confuse that reason is too large log or too busy disk I/O #296

Closed
wants to merge 2 commits into from

Conversation

@asdf2014
Copy link
Member

@asdf2014 asdf2014 commented Jun 28, 2017

FileChannel#size info should be added to FileTxnLog#commit to solve the confuse that reason is too large log or too busy disk I/O


Example from jenkins log:

2017-06-28 10:25:16,645 [myid:] - WARN  [SyncThread:0:FileTxnLog@341] - fsync-ing the write ahead log (1945616 bytes) in SyncThread:0 took 2240ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
…g#commit` to solve the confuse that reason is too large log or too busy disk I/O

syncElapsedMS = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startSyncNS);
if (syncElapsedMS > fsyncWarningThresholdMS) {
LOG.warn("fsync-ing the write ahead log in "
LOG.warn("fsync-ing the write ahead log ("
+ channel.size() + " bytes) in "

This comment has been minimized.

@afine

afine Jun 29, 2017
Contributor

According to the javadoc, size() returns "Returns the current size of this channel's file." Can you explain why this is valuable?

This comment has been minimized.

@asdf2014

asdf2014 Jun 30, 2017
Author Member

@afine Thank you for code review. When we got the warn message, will confuse that reason is too large log or too busy disk I/O? So, need to show the size of log to eliminate the too large log possibility.

This comment has been minimized.

@phunt

phunt Jan 19, 2018
Contributor

@asdf2014 can you move the changed text after the first sentence?

... latency. File size is ### bytes. See ...

The reason being that I know a number of users who have log triggers on this log message and I'd like to minimize the impact as much as possible.

This comment has been minimized.

@afine

afine Jan 30, 2018
Contributor

@asdf2014 as soon as @phunt's comment is addressed I can merge this in.

@asdf2014
Copy link
Member Author

@asdf2014 asdf2014 commented Dec 22, 2017

@afine Could you approve this PR?
@phunt @anmolnar PTAL.


Just recently, i experienced this situation again, this time due to the bad disk rather than the huge WAL file. But i still cannot get the root cause from Zookeeper log...
image

@anmolnar
Copy link
Contributor

@anmolnar anmolnar commented Jan 11, 2018

Hi @asdf2014

I'm happy with the change if it has any help to you for debugging.

@asdf2014
Copy link
Member Author

@asdf2014 asdf2014 commented Jan 12, 2018

Hi, @anmolnar . Thanks, would you approve this PR?


syncElapsedMS = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startSyncNS);
if (syncElapsedMS > fsyncWarningThresholdMS) {
LOG.warn("fsync-ing the write ahead log in "
LOG.warn("fsync-ing the write ahead log ("
+ channel.size() + " bytes) in "

This comment has been minimized.

@afine

afine Jan 30, 2018
Contributor

@asdf2014 as soon as @phunt's comment is addressed I can merge this in.

@asdf2014
Copy link
Member Author

@asdf2014 asdf2014 commented Jan 31, 2018

@phunt @afine Thank you for those suggestions. Considered very thoughtful. Already fixed.

@afine
afine approved these changes Feb 1, 2018
@asfgit asfgit closed this in 069c3e4 Feb 1, 2018
asfgit pushed a commit that referenced this pull request Feb 1, 2018
…g#commit` to solve the confuse that reason is too large log or too busy disk I/O

`FileChannel#size` info should be added to `FileTxnLog#commit` to solve the confuse that reason is too large log or too busy disk I/O

------
Example from jenkins [log](https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/830/console):
```java
2017-06-28 10:25:16,645 [myid:] - WARN  [SyncThread:0:FileTxnLog341] - fsync-ing the write ahead log (1945616 bytes) in SyncThread:0 took 2240ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
```

Author: asdf2014 <benedictjin2016@gmail.com>
Author: asdf2014 <1571805553@qq.com>

Reviewers: Patrick Hunt <phunt@apache.org>, Andor Molnár <andor@cloudera.com>, Abraham Fine <afine@apache.org>

Closes #296 from asdf2014/ZOOKEEPER-2824 and squashes the following commits:

7cf4b68 [asdf2014] Improve latency log description
dd2268f [asdf2014] ZOOKEEPER-2824: `FileChannel#size` info should be added to `FileTxnLog#commit` to solve the confuse that reason is too large log or too busy disk I/O

(cherry picked from commit 069c3e4)

Signed-off-by: Abraham Fine <afine@apache.org>

Signed-off-by: Abraham Fine <afine@apache.org>
@afine
Copy link
Contributor

@afine afine commented Feb 1, 2018

Merged. Thanks @asdf2014!

@asdf2014
Copy link
Member Author

@asdf2014 asdf2014 commented Feb 2, 2018

@afine You are welcome :-)

@asdf2014 asdf2014 deleted the asdf2014:ZOOKEEPER-2824 branch Feb 2, 2018
lvfangmin pushed a commit to lvfangmin/zookeeper that referenced this pull request Jun 17, 2018
…g#commit` to solve the confuse that reason is too large log or too busy disk I/O

`FileChannel#size` info should be added to `FileTxnLog#commit` to solve the confuse that reason is too large log or too busy disk I/O

------
Example from jenkins [log](https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/830/console):
```java
2017-06-28 10:25:16,645 [myid:] - WARN  [SyncThread:0:FileTxnLog341] - fsync-ing the write ahead log (1945616 bytes) in SyncThread:0 took 2240ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
```

Author: asdf2014 <benedictjin2016@gmail.com>
Author: asdf2014 <1571805553@qq.com>

Reviewers: Patrick Hunt <phunt@apache.org>, Andor Molnár <andor@cloudera.com>, Abraham Fine <afine@apache.org>

Closes apache#296 from asdf2014/ZOOKEEPER-2824 and squashes the following commits:

7cf4b68 [asdf2014] Improve latency log description
dd2268f [asdf2014] ZOOKEEPER-2824: `FileChannel#size` info should be added to `FileTxnLog#commit` to solve the confuse that reason is too large log or too busy disk I/O
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants