Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[reporter] Stuck after reporter Run is canceled by context #43

Closed
at15 opened this issue Mar 15, 2018 · 2 comments
Closed

[reporter] Stuck after reporter Run is canceled by context #43

at15 opened this issue Mar 15, 2018 · 2 comments
Labels
bug
Projects
Runner Reporter
Milestone
Week 10

Comments

@at15
Copy link
Member

at15 commented Mar 15, 2018 •
edited
Loading

got stuck before finalize reporter, for counter reporter, there is nothing to finalize, should be some go routine problem on my end ...
@at15 at15 added the bug label Mar 15, 2018
@at15
Copy link
Member Author

at15 commented Mar 19, 2018

met it again when using akumuli ...

INFO 0000 target database is akumuli_0 type akumuli
INFO 0000 workload is workload_0 series 1 value generator is constant
INFO 0000 reporter counter_0 is type counter
INFO 0000 counter reporter start 2018-03-18 18:50:22.688795797 -0700 PDT m=+0.022608676
INFO 0000 worker 3 started
INFO 0000 worker 7 started
INFO 0000 worker 6 started
INFO 0000 worker 4 started
INFO 0000 worker 8 started
INFO 0000 worker 0 started
INFO 0000 worker 9 started
INFO 0000 worker 5 started
INFO 0000 worker 1 started
INFO 0000 worker 2 started
INFO 0010 worker 8 exit due to context
INFO 0010 worker 0 exit due to context
INFO 0010 worker 2 exit due to context
INFO 0010 worker 3 exit due to context
INFO 0010 worker 9 exit due to context
INFO 0010 counter reporter stopped by context
INFO 0010 counter reporter stop 2018-03-18 18:50:32.689397978 -0700 PDT m=+10.023210745 duration 10.000602069s

at15 added a commit that referenced this issue Mar 19, 2018
@at15
[runner] Use separated context for reporter #43
5d704da 
- reporter need to keep draining the result channel, previously it is
canceled along with workers in context, it's pretty likely it is
canceled when some worker are still sending result, thus the
`wg.Wait()` never exits because worker are sending to a channel that is
filled and never get drained ....
- it was not happending very often because the number of concurrent
worker is small and server response time is large ....
at15 added a commit that referenced this issue Mar 19, 2018
@at15
[runner] Add wait group for reporter #43
f123c58 
- was getting negative duration due to finialize is called before Run
is finished, another way to solve this is to add a lock inside reporter, so
calling Finalize won't work until Run exits ...
@at15
Copy link
Member Author

at15 commented Mar 19, 2018

solved in #45 , need to keep in mind when using both context can channel, the reader should not use same context as writer, otherwise, the writer are pretty likely to be blocked ....

@at15 at15 closed this as completed Mar 19, 2018
@at15 at15 added this to To do in Runner via automation Mar 19, 2018
@at15 at15 added this to To do in Reporter via automation Mar 19, 2018
@at15 at15 added this to the Week 10 milestone Mar 19, 2018
@at15 at15 moved this from To do to Done in Reporter Mar 19, 2018
@at15 at15 moved this from To do to Done in Runner Mar 19, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
Reporter
  
Done
Runner
  
Done
Milestone
Week 10
Development

No branches or pull requests
1 participant
@at15