Issue #481 fix hanging race condition #482

ColinDKelley · 2020-08-12T04:30:29Z

Fixes issue #481 by removing code that attempted to synchronize threads with sleep() and Thread#wakeup. That was a race condition bug since if wakeup ran before sleep(), it would be lost and the sleeping thread would simply hang.

Detailed changes:

Use ::Mutex/::ConditionVariable to wait/wake up on state transitions. I factored this down into FSM since it's used in two state machine classes now.
Clean up and simplify FSM. Enforce symbol contracts, remove unnecessary methods, rename default_state to start_state since the latter is standard FSM terminology. Also initialize with initialize_fsm vs super().
Use Queue#pop to block front-end when waiting for events.
Combine Loop#setup and Loop#resume since they were always called consecutively. This allowed the Listener :front_end_ready state to disappear.
Remove Loop#resume and Loop#pause since they were no-ops.
Rename teardown to stop to be consistent. (At some point I'd like to do a similar thing with :processing_events. This could simply be called :started.)

lib/listen/listener.rb

lib/listen/fsm.rb

lib/listen/event/loop.rb

coveralls · 2020-08-12T04:40:24Z

Coverage decreased (-0.7%) to 97.026% when pulling aabfbbf on Invoca:issue_481-fix-hanging-race-condition into ba5059c on guard:master.

spec/lib/listen/adapter/linux_spec.rb

lib/listen/event/loop.rb

lib/listen/event/config.rb

ioquatix

Thanks for your effort. I have asked for some changes.

In the future, please make smaller PRs focusing on one particular issue.

Pay attention to changes to public interfaces since this can cause a lot of pain downstream.

I look forward to reviewing and merging this PR and others from you.

lib/listen/event/loop.rb

ioquatix · 2020-08-23T20:47:16Z

lib/listen/event/loop.rb

      class Error < RuntimeError
-        class NotStarted < Error
-        end
+        class ThreadFailedToStart < Error; end


Since NotStarted is a public interface, please reuse it instead of introducing ThreadFailedToStart.

Good point--I hadn't thought about it being in the public interface. I will put that back to its old name.

I restored it to its old name, NotStarted. And if start is called when we're already started, I put the code back as it was to simply return as a no-op, since there could be callers who were counting on that public behavior.
0b236c1

ioquatix · 2020-08-23T20:48:36Z

lib/listen/event/loop.rb

+
+      def start
+        transition! :starting do
+          state == :pre_start or raise Error::AlreadyStarted


I would say it's better to use explicit if statement here

unless state == :pre_start raise Error::AlreadyStarted... end

ioquatix · 2020-08-23T20:48:57Z

lib/listen/event/loop.rb

-        class NotStarted < Error
-        end
+        class ThreadFailedToStart < Error; end
+        class AlreadyStarted < Error; end


I wonder if this should inherit from NotStarted to preserve existing semantics where possible.

That might be a bit confusing though?

AlreadyStarted.is_a?(NotStarted)

I will look into what was raised before if start was called a second time.

Yes, understood. In this context... if someone calls start, and it's already started, that's a kind of NotStarted because it was already started. If you can think of something better, it's worth considering.

Restored to its old behavior: 0b236c1

lib/listen/fsm.rb

ioquatix · 2020-08-23T20:51:59Z

lib/listen/fsm.rb

+      # Passing a state name sets the start state
+      def start_state(new_start_state = nil)
+        if new_start_state
+          new_start_state.is_a?(Symbol) or raise ArgumentError, "state name must be a Symbol (got #{new_start_state.inspect})"


Instead of explicitly detecting it's a symbol, why not use to_sym?

This is a style we like for enforcing contracts, especially around symbol-vs-string! There aren't any callers of this method passing strings, so .to_sym would be a no-op now. Adding .to_sym now might serve to enable more confusing usage later...but there's no need for that, is there?

This kind of pseudo type checking is uncommon in Ruby, even thought I can see how it makes sense.

The reality is, if you add this, there are many other places where you might start trying to enforce type contracts.

Agreed, we don't really want users passing in strings.

True that it's not common in Ruby (yet). But we've found it to be invaluable for containing String vs. Symbol insanity! Otherwise, you wind up with defensive code sprinkled everywhere that tries to handle String and Symbol as equivalent...and you never know if that clutter is complete.

Speaking of which, I added the missing spec for the FSM module and made sure to test this case: bbd10ee

ioquatix · 2020-08-23T20:53:53Z

lib/listen/fsm.rb

+    # Low-level, immediate state transition with no checks or callbacks.
+    def transition!(new_state_name)
+      new_state_name.is_a?(Symbol) or raise ArgumentError, "state name must be a Symbol (got #{new_state_name.inspect})"
+      @mutex or raise ArgumentError, "FSM not initialized. You must call super() from initialize!"


This seems like an overly specific issue to check for - I can imagine some code was not working correctly, but in that case @mutex.synchronize will blow up with nil error... do we need to manually check it?

I added this because of the contract that the including code must call super() first. That's pretty unusual for mixins, but it's how the code was already structured, and it work well in this case. It's critical to get the mutex initialized up front because the lazy style that's often used in mixins (@mutex ||= Mutex.new) wouldn't work; ironically, it would need a synchronizing mutex around it!

Ah, this might make it more clear: how about an overall flag like @_fsm_initialized that's set = true when super() is called. This could code check that for the ArgumentError test. If that test passes, we can be confident that @mutex has been initialized.

Understood.

Well, I don't really want to make the code any more complicated than it already is.

If the code is not implemented correctly (missing call to super) it's a mechanical problem - i.e. the code needs to be fixed. This isn't a transient run-time issue. The current implementation is okay, but it's also very explicit in a non-canonical way.

I don't think we should be adding extra instance variables to enforce initialisation.

I realized we had to move away from super(). There's a reason I've never seen mixins initialized that way before: it precludes any base class initialization, since the mixin would have to do that for you (by calling super itself), but would have no way to know the base class's initialize signature!

Instead I went with an explicitly-named method initialize_fsm. And stored a variable to remember if that didn't happen. And I addressed the longstanding comment in there to make the FSM methods private. And I added a spec for the FSM module and tested all of this. I think it's all much cleaner now! LMK what you think.

ioquatix · 2020-08-23T20:55:03Z

lib/listen/fsm.rb

        @name = name
        @block = block
-        @transitions = nil
-        @transitions = Array(transitions).map(&:to_sym) if transitions
+        @transitions = if transitions


You don't need if transitions.

I kept it from the old code. You can see it hiding up there one line up... in "trailing if" form. There is code below that counts on it. It's just after this comment:

All transitions are allowed if none are expressly declared

It's simply not needed. Array(nil) => [].

I don't follow. That if is there so that when transitions is nil, @transitions will be as well. The code below counts on that, because it uses nil in @transitions as an indicator that all transitions are allowed.

ioquatix · 2020-08-23T22:05:44Z

You may need to rebase on master.

ColinDKelley

Thanks for the review, @ioquatix.

I've responded to all the comments you left. 3 of them require changes. I'll try to push those up in the next day or two.

ColinDKelley · 2020-08-23T21:26:15Z

lib/listen/event/loop.rb

      class Error < RuntimeError
-        class NotStarted < Error
-        end
+        class ThreadFailedToStart < Error; end


Good point--I hadn't thought about it being in the public interface. I will put that back to its old name.

ColinDKelley · 2020-08-23T21:27:59Z

lib/listen/event/loop.rb

-        class NotStarted < Error
-        end
+        class ThreadFailedToStart < Error; end
+        class AlreadyStarted < Error; end


That might be a bit confusing though?

AlreadyStarted.is_a?(NotStarted)

I will look into what was raised before if start was called a second time.

ColinDKelley · 2020-08-23T21:29:41Z

lib/listen/event/loop.rb

+
+      def start
+        transition! :starting do
+          state == :pre_start or raise Error::AlreadyStarted


We're fans for the one-line and/or form for concise preconditions [this use case is exactly why and and or are in Ruby, from Perl], but I can certainly change it to the 3-line form.

lib/listen/fsm.rb

ColinDKelley · 2020-08-23T21:41:35Z

lib/listen/fsm.rb

+      # Passing a state name sets the start state
+      def start_state(new_start_state = nil)
+        if new_start_state
+          new_start_state.is_a?(Symbol) or raise ArgumentError, "state name must be a Symbol (got #{new_start_state.inspect})"


This is a style we like for enforcing contracts, especially around symbol-vs-string! There aren't any callers of this method passing strings, so .to_sym would be a no-op now. Adding .to_sym now might serve to enable more confusing usage later...but there's no need for that, is there?

ColinDKelley · 2020-08-23T21:46:16Z

lib/listen/fsm.rb

+    # Low-level, immediate state transition with no checks or callbacks.
+    def transition!(new_state_name)
+      new_state_name.is_a?(Symbol) or raise ArgumentError, "state name must be a Symbol (got #{new_state_name.inspect})"
+      @mutex or raise ArgumentError, "FSM not initialized. You must call super() from initialize!"


I added this because of the contract that the including code must call super() first. That's pretty unusual for mixins, but it's how the code was already structured, and it work well in this case. It's critical to get the mutex initialized up front because the lazy style that's often used in mixins (@mutex ||= Mutex.new) wouldn't work; ironically, it would need a synchronizing mutex around it!

Ah, this might make it more clear: how about an overall flag like @_fsm_initialized that's set = true when super() is called. This could code check that for the ArgumentError test. If that test passes, we can be confident that @mutex has been initialized.

ColinDKelley · 2020-08-23T21:49:22Z

lib/listen/fsm.rb

        @name = name
        @block = block
-        @transitions = nil
-        @transitions = Array(transitions).map(&:to_sym) if transitions
+        @transitions = if transitions


I kept it from the old code. You can see it hiding up there one line up... in "trailing if" form. There is code below that counts on it. It's just after this comment:

All transitions are allowed if none are expressly declared

spec/lib/listen/fsm_spec.rb

houndci-bot · 2020-08-24T17:10:47Z

spec/lib/listen/fsm_spec.rb

@@ -0,0 +1,103 @@
+RSpec.describe Listen::FSM do


Style/FrozenStringLiteralComment: Missing magic comment # frozen_string_literal: true.

I will make a separate PR for this later.

spec/lib/listen/fsm_spec.rb

spec/lib/listen/adapter/polling_spec.rb

lib/listen/fsm.rb

lib/listen/event/loop.rb

lib/listen/file.rb

spec/lib/listen/file_spec.rb

ColinDKelley · 2020-08-27T21:33:44Z

@ioquatix I believe all comments are addressed here. I merged up latest master as well. There were a couple build failures that came across with master. I sent a separate PR #488 for those and merged it in here.

ColinDKelley · 2020-08-31T17:50:51Z

Hi @ioquatix
Anything else to do on this PR?

ioquatix · 2020-09-07T02:52:53Z

My apologies, I was busy for Ruby Kaigi conference. I'm reviewing it now.

ioquatix · 2020-09-07T02:58:35Z

I'll fix the Ruby 3 issues. However can you check why:

Failures:

  1) Listen with one listen dir force_polling option to false with default ignore options two dirs with files in listen dir listens to dir move
     Failure/Error:
       expect(wrapper.listen do
         mv 'dir1', 'dir2/'
       end).to eq expected

       expected: {:added=>["dir2/dir1/file1.rb"], :modified=>[], :removed=>["dir1/file1.rb"]}
            got: {:added=>["dir2/dir1/file1.rb"], :modified=>["dir2/file2.rb"], :removed=>["dir1/file1.rb"]}

       (compared using ==)

ioquatix · 2020-09-07T04:20:27Z

Okay, I see a bunch of specs are failing, I'll try to bring master back to green.

ioquatix · 2020-09-07T05:34:10Z

Okay, master is green again, can you rebase?

ColinDKelley · 2020-09-12T23:43:18Z

@ioquatix Thanks for getting master green again. I've merged that in and now this branch is green too!

ioquatix · 2020-09-14T03:25:44Z

Can you please rebase on master so I can merge - please don't merge master into your branch because it creates a huge mess.

… :started, :stopped

…tops the suite)

… sleep()/wakeup

…lock

…fault_state to start_state

…there are multiple state machines)

…) instead of _sleep

… instance method that just delegate to class, require symbols

…ure multiple threads are waiting

… called in @fsm_initialized

…hods private

…tStarted), and silently no-op on start if already started

…d to be raised

spec/support/acceptance_helper.rb

spec/lib/listen/fsm_spec.rb

spec/lib/listen/event/processor_spec.rb

lib/listen/fsm.rb

ColinDKelley · 2020-09-14T23:19:00Z

@ioquatix : Rebased and pushed.

ioquatix · 2020-09-16T04:17:52Z

Great work, thanks for your effort. Do you have time to jump on call some time this week to discuss a release plan/schedule?

ColinDKelley · 2020-09-17T04:56:12Z

Hi @ioquatix Thank you! Yes, this week is good for a call. I'm on US/Pacific time. How about if we pick an exact time and medium by email?

houndci-bot reviewed Aug 12, 2020

View reviewed changes

houndci-bot reviewed Aug 13, 2020

View reviewed changes

ioquatix requested changes Aug 23, 2020

View reviewed changes

ColinDKelley commented Aug 23, 2020

View reviewed changes

houndci-bot reviewed Aug 24, 2020

View reviewed changes

houndci-bot reviewed Aug 27, 2020

View reviewed changes

lib/listen/event/loop.rb Outdated Show resolved Hide resolved

lib/listen/event/loop.rb Show resolved Hide resolved

ColinDKelley force-pushed the issue_481-fix-hanging-race-condition branch 2 times, most recently from 4a0458a to d7efb56 Compare August 27, 2020 19:17

houndci-bot reviewed Aug 27, 2020

View reviewed changes

lib/listen/file.rb Outdated Show resolved Hide resolved

lib/listen/file.rb Outdated Show resolved Hide resolved

lib/listen/file.rb Outdated Show resolved Hide resolved

ColinDKelley force-pushed the issue_481-fix-hanging-race-condition branch from d7efb56 to 17c612d Compare August 27, 2020 19:22

houndci-bot reviewed Aug 27, 2020

View reviewed changes

spec/lib/listen/file_spec.rb Outdated Show resolved Hide resolved

ioquatix mentioned this pull request Sep 8, 2020

Allow Ruby 3 #490

Merged

ColinDKelley force-pushed the issue_481-fix-hanging-race-condition branch from 92a532c to b4c5afe Compare September 12, 2020 23:40

ColinDKelley added 6 commits September 14, 2020 13:42

issue guard#481: rename setup to start; states :pre_start, :starting,…

de68bbe

… :started, :stopped

issue guard#481: rename _wait_for_changes _process_changes

710b092

issue guard#481: start now starts the thread (no need for resume)

5709e6b

issue guard#481: collapse :frontend_ready state into :processing_events

4a3e6f5

issue guard#481: raise error instead of abort (since abort abruptly s…

92c16e4

…tops the suite)

issue guard#481: use queue.pop to block background thread rather than…

e76a7c8

… sleep()/wakeup

ColinDKelley added 20 commits September 14, 2020 13:58

issue guard#481: close queue on stop; move usable? to class << self b…

84401ef

…lock

issue guard#481: remove &block from queue

b8f7e42

issue guard#481: use __dir__ instead of __FILE__

26a5aa8

issue guard#481: move Mutex and ConditionVariable into FSM; rename de…

5cde98c

…fault_state to start_state

issue guard#481: rename Loop#teardown (aka processor) to #stop

0d24537

issue guard#481: expose listener so state checks are explicit (since …

2b03f3e

…there are multiple state machines)

issue guard#481: listener.wait_for_state(:processing_events, :stopped…

dd7b4c6

…) instead of _sleep

issue guard#481: _sleep now always takes seconds

cdde9bd

issue guard#481: remove unused &block

867ba98

issue guard#481: dynamically compute non-paused Listener states

8ae20de

issue guard#481: clean up FSM: call super, omit unused defaults, omit…

b99f794

… instance method that just delegate to class, require symbols

issue guard#481: use broadcast rather than signal, in case in the fut…

fedf79a

…ure multiple threads are waiting

issue guard#481: fix linux tests

2366e4a

issue guard#481: rename FSM#initialize to initialize_fsm; remember if…

dcc28d5

… called in @fsm_initialized

issue guard#481: allow queue.close; remove test of wakeup_on_event

0c5989d

issue guard#481: drop tests for wakeup_on_event; expect .join.kill

f90e882

issue guard#481: add spec for Listen::FSM; address TODO by making met…

67c4053

…hods private

issue guard#481: appease hound

c3dd34c

issue guard#481: restore previous public behavior: exception name (No…

794571c

…tStarted), and silently no-op on start if already started

issue guard#481: fix duplicate start test to not expect AlreadyStarte…

24dec52

…d to be raised

ColinDKelley force-pushed the issue_481-fix-hanging-race-condition branch from b4c5afe to 24dec52 Compare September 14, 2020 22:59

houndci-bot reviewed Sep 14, 2020

View reviewed changes

ioquatix merged commit 86e06ae into guard:master Sep 16, 2020

ColinDKelley deleted the issue_481-fix-hanging-race-condition branch October 30, 2020 21:37

Issue #481 fix hanging race condition #482

Issue #481 fix hanging race condition #482

Conversation

ColinDKelley commented Aug 12, 2020 • edited Loading

coveralls commented Aug 12, 2020 • edited Loading

ioquatix left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ColinDKelley Aug 23, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ioquatix commented Aug 23, 2020

ColinDKelley left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ColinDKelley Aug 23, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ColinDKelley commented Aug 27, 2020

ColinDKelley commented Aug 31, 2020

ioquatix commented Sep 7, 2020

ioquatix commented Sep 7, 2020

ioquatix commented Sep 7, 2020

ioquatix commented Sep 7, 2020

ColinDKelley commented Sep 12, 2020 • edited Loading

ioquatix commented Sep 14, 2020

ColinDKelley commented Sep 14, 2020

ioquatix commented Sep 16, 2020

ColinDKelley commented Sep 17, 2020

ColinDKelley commented Aug 12, 2020 •

edited

Loading

coveralls commented Aug 12, 2020 •

edited

Loading

ColinDKelley Aug 23, 2020 •

edited

Loading

ColinDKelley left a comment •

edited

Loading

ColinDKelley Aug 23, 2020 •

edited

Loading

ColinDKelley commented Sep 12, 2020 •

edited

Loading