Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Improve debug messages in Voldemort #89

Merged
merged 1 commit into from Aug 13, 2012

Conversation

Projects
None yet
2 participants
Contributor

pbailis commented Aug 9, 2012

This patch improves the debugging messages introduced by commit fa3d6f6 in two ways:
1.) Print out hex representation of keys and values instead of current array format ("[48, 49, 79]").
2.) In tracing the stack traces, use the inputStream instead of the request handler; because the streamRequestHandler is used between requests, so using its identityHashCode causes collisions. Each DataInputStream is unique between requests, so we now use it instead.

@zhongjiewu zhongjiewu added a commit that referenced this pull request Aug 13, 2012

@zhongjiewu zhongjiewu Merge pull request #89 from pbailis/master
Improve debug messages in Voldemort
c8dcd82

@zhongjiewu zhongjiewu merged commit c8dcd82 into voldemort:master Aug 13, 2012

Collaborator

zhongjiewu commented Aug 17, 2012

Hi Peter,
The hudson is failing for Hinted Handoff test all the time. This test may fail from time to time but not all the time usually. I need to roll it back and test it locally before putting your changes back again. Did you remember if tests with debug on and off both passes?

@ghost ghost assigned zhongjiewu Aug 17, 2012

Contributor

pbailis commented Aug 17, 2012

IIRC, junit-long passed on my local machine.

Which test is the one that's failing? I just reran ant junit-test -Dtest.name=voldemort.store.routed.HintedHandoffTest on my local box with DEBUG statements both enabled and disabled and it passes in both cases.

Collaborator

zhongjiewu commented Aug 17, 2012

on Hudson, Job 74,75,76 that is tested after the merge is failing. I reverted it for now to prevent blocking the trunk. I think we should work on a branch and run more hudson jobs on the branch to make sure things work. I think I will probably see that same result if I run these two tests. I admit that this test is very funky.
My hypothesis is that your extra code is changing the time of how the test runs and make the failing condition reproduce every time it runs.

Collaborator

zhongjiewu commented Aug 17, 2012

Contributor

pbailis commented Aug 17, 2012

I'm not able to repro this on my local machine.

The failure message doesn't seem related to my changes:
data should be stored in the slop for key = [7, -62, -7, 23, 118, 126, -103, -120, -31, 77, -36, 51, -72, 52, -98, -89, 32, -91, -92, 115, -6, 79, 73, -70, 119, -35, -61, -24, 80, -21, 22, -2]
(from http://test.project-voldemort.com:8080/job/voldemort-master/76/)

It's possible this is a timing issue. I'm curious if you can reproduce on your local machine with ant junit-test -Dtest.name=voldemort.store.routed.HintedHandoffTest

Collaborator

zhongjiewu commented Aug 17, 2012

I could reproduce it on my machine, both alone and inside junit.
However it does not fail every time.
This test uses src/java/voldemort/store/routed/action/ files, which some of your changes touch.
I am occupied by a prod-related feature change so I will try to see if such failure can be explained with the old explanation.

Collaborator

zhongjiewu commented Aug 17, 2012

I meant it does not fail every time when run alone. But never passed in "junit" task

Collaborator

zhongjiewu commented Aug 17, 2012

OK, the whole junit is also kind of unstable in results.

Contributor

pbailis commented Aug 26, 2012

Zhongjie--did we figure this out?

Collaborator

zhongjiewu commented Aug 27, 2012

I need to pull it and run on eclipse myself and see what problem it is.

Collaborator

zhongjiewu commented Aug 27, 2012

Hi Peter, can you configure your Hudson job to run test on your branch that you want to pull? If that passes, then things will be a little weird.

Collaborator

zhongjiewu commented Aug 28, 2012

Interesting. I saw successful Hudson build for your branch.

Collaborator

zhongjiewu commented Aug 28, 2012

Your repo is behind quite a few commits as far as I can see. This could be why test results are not consistent.
Can you do a rebase from voldemort master? Don't include my revert. Make sure your pull request part does not get reverted and disappeared.

Contributor

pbailis commented Aug 28, 2012

I'll do this tomorrow, though the fact that my Hudson job passed hints that this isn't a(n immediate) problem with my pull request.

Collaborator

zhongjiewu commented Aug 28, 2012

I have a theory.
So your changes in the HintedHandOff is exposing the test bug. I run the tests that excludes your change in HintedHandOff and it works fine.
So I will take a look at why your change is making that test fail everytime.
Though I know that this is not your fault.

Collaborator

zhongjiewu commented Aug 28, 2012

Hey Peter, is the logging changes in HintedHandoff Important for you?
We can try commiting everything except that file and see if the tests passes.

Contributor

pbailis commented Aug 30, 2012

We can roll back the HintedHandoff changes--it's just for prettyprinting the key names.

Collaborator

zhongjiewu commented Sep 6, 2012

Just let you know, I have put your changes back. They are now in the master(except Hinted handoff file) and has been passing tests.
I am doing a release and they will be release in a week at most(hopefully done today).

Contributor

pbailis commented Sep 7, 2012

Awesome--thanks! I'm now back in the country and should be faster to reply.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment