Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix a race condition preventing embedded compiler to shutdown after a protocol error #2106

Merged
merged 3 commits into from
Oct 5, 2023
Merged

Fix a race condition preventing embedded compiler to shutdown after a protocol error #2106

merged 3 commits into from
Oct 5, 2023

Conversation

ntkme
Copy link
Contributor

@ntkme ntkme commented Oct 4, 2023

There is an extreme rare race condition during compiler shutdown after a protocol error is send from host callable here:

dispatcher.sendError(handleError(error, stackTrace));

A reproduction looks like this:

require 'sass-embedded'

def try
  puts yield
rescue StandardError => e
  puts e
end

fn = lambda do |_args|
  # this is an invalid calculation that will trigger a protocol error
  Sass::Value::Calculation.send(:new, 'foo', [Sass::Value::Number.new(1)])
end

try do
  Sass.compile_string('a {b: foo()}',
                      functions: { 'foo()': fn }).css
end

try do
  Sass.compile_string('a {b: c}').css
end

The fundamental problem here is that StreamSink#close() is async operation which can take time if the system is slow. This gives us a chance to send a second message before compiler has been shutdown.

The sequence of events looks like:

  1. Host send first request that triggers a protocol error in host callable response.
  2. Compiler send ProtocolError to host, however it forget to set _requestError = true in this case.
  3. Host send another request, and this request would be accepted because nothing has been cleaned up yet.
  4. Because _requestError is not set, the thrown ProtocolError bubbles up as a CompileFailure.
  5. Compiler send CompileFailure for the first request, and this would set _checkedOut = false on the isolate.
  6. Compiler send CompilerSuccess for the second request, and get Unhandled exception: Bad state: Shouldn't receive a message before being checked out.

The compiler would get stuck after the unhandled exception. The fix here is to make sure _requestError = true is set to that it does not send an extra CompilerFailure later which would reset isolate state at incorrect timing.

Slow sink close can be emulated with the following patch to make it easy to reproduce:

diff --git a/lib/src/embedded/isolate_dispatcher.dart b/lib/src/embedded/isolate_dispatcher.dart
index 044e2b9c..b58732f6 100644
--- a/lib/src/embedded/isolate_dispatcher.dart
+++ b/lib/src/embedded/isolate_dispatcher.dart
@@ -118,7 +118,9 @@ class IsolateDispatcher {
         // isolate, so we just send them as-is and close out the underlying
         // channel.
         sendError(compilationId, error);
-        _channel.sink.close();
+        Timer(Duration(milliseconds: 100), () =>
+          _channel.sink.close()
+        );
       } else {
         _handleError(error, stackTrace);
       }

Copy link
Contributor

@nex3 nex3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

CHANGELOG.md Outdated Show resolved Hide resolved
@nex3 nex3 merged commit 310904e into sass:main Oct 5, 2023
44 checks passed
@ntkme ntkme deleted the fix-shutdown-race-condition branch October 5, 2023 00:40
jgerigmeyer added a commit to oddbird/dart-sass that referenced this pull request Oct 12, 2023
* feature.color-4: (30 commits)
  Update for `lch()` tests (sass#2108)
  Add support for relative color syntax (sass#2112)
  Update for `oklab()` tests (sass#2094)
  Poke CI
  Cut a release (sass#2107)
  Implement first class mixins (sass#2073)
  Fix a race condition preventing embedded compiler to shutdown after a protocol error (sass#2106)
  Switch to the GitHub-hosted MacOS ARM64 runner (sass#2103)
  Update the version of Sass used by the website on release (sass#2102)
  Bump actions/checkout from 3 to 4 (sass#2088)
  Bump docker/setup-qemu-action from 2 to 3 (sass#2089)
  Implement support for the relative color syntax of CSS Color 5 (sass#2098)
  Rephrase errors for numbers that must be unitless or % (sass#2101)
  Forbid LLM contributions (sass#2100)
  Update protocol-version during embedded-host-node release (sass#2097)
  Deprecate Deprecation.calcInterp (sass#2096)
  Avoid useless allocations for interpolations without maps (sass#2095)
  Fix an error during embedded compiler shutdown (sass#2092)
  Cut a release (sass#2090)
  Expose the containing URL to importers under some circumstances (sass#2083)
  ...
Friendly-users added a commit to Friendly-users/dart-sass that referenced this pull request Jun 28, 2024
-----
It is inappropriate to include political and offensive content in public code repositories.

Public code repositories should be neutral spaces for collaboration and community, free from personal or political views that could alienate or discriminate against others. Political content, especially that which targets or disparages minority groups, can be harmful and divisive. It can make people feel unwelcome and unsafe, and it can create a hostile work environment.

Please refrain from adding such content to public code repositories.
---

sass#2000 sass#2001 sass#2002 sass#2003 sass#2004 sass#2005 sass#2006 sass#2007 sass#2008 sass#2009 sass#2010 sass#2011 sass#2012 sass#2013 sass#2014 sass#2015 sass#2016 sass#2017 sass#2018 sass#2019 sass#2020 sass#2021 sass#2022 sass#2023 sass#2024 sass#2025 sass#2026 sass#2027 sass#2028 sass#2029 sass#2030 sass#2031 sass#2032 sass#2033 sass#2034 sass#2035 sass#2036 sass#2037 sass#2038 sass#2039 sass#2040 sass#2041 sass#2042 sass#2043 sass#2044 sass#2045 sass#2046 sass#2047 sass#2048 sass#2049 sass#2050 sass#2051 sass#2052 sass#2053 sass#2054 sass#2055 sass#2056 sass#2057 sass#2058 sass#2059 sass#2060 sass#2061 sass#2062 sass#2063 sass#2064 sass#2065 sass#2066 sass#2067 sass#2068 sass#2069 sass#2070 sass#2071 sass#2072 sass#2073 sass#2074 sass#2075 sass#2076 sass#2077 sass#2078 sass#2079 sass#2080 sass#2081 sass#2082 sass#2083 sass#2084 sass#2085 sass#2086 sass#2087 sass#2088 sass#2089 sass#2090 sass#2091 sass#2092 sass#2093 sass#2094 sass#2095 sass#2096 sass#2097 sass#2098 sass#2099 sass#2100 sass#2101 sass#2102 sass#2103 sass#2104 sass#2105 sass#2106 sass#2107 sass#2108 sass#2109 sass#2110 sass#2111 sass#2112 sass#2113 sass#2114 sass#2115 sass#2116 sass#2117 sass#2118 sass#2119 sass#2120 sass#2121 sass#2122 sass#2123 sass#2124 sass#2125 sass#2126 sass#2127 sass#2128 sass#2129 sass#2130 sass#2131 sass#2132 sass#2133 sass#2134 sass#2135 sass#2136 sass#2137 sass#2138 sass#2139 #2140 sass#2141 sass#2142 sass#2143 sass#2144 sass#2145 sass#2146 sass#2147 sass#2148 sass#2149 sass#2150 sass#2151 sass#2152 sass#2153 sass#2154 sass#2155 sass#2156 sass#2157 sass#2158 sass#2159 sass#2160 sass#2161 sass#2162 sass#2163 sass#2164 sass#2165 sass#2166 sass#2167 sass#2168 sass#2169 sass#2170 sass#2171 sass#2172 sass#2173 sass#2174 sass#2175 sass#2176 sass#2177 sass#2178 sass#2179 sass#2180 sass#2181 sass#2182 sass#2183 sass#2184 sass#2185 sass#2186 sass#2187 sass#2188 sass#2189 sass#2190 sass#2191 sass#2192 sass#2193 sass#2194 sass#2195 sass#2196 sass#2197 sass#2198 sass#2199 sass#2200 sass#2201 sass#2202 sass#2203 sass#2204 sass#2205 sass#2206 sass#2207 sass#2208 sass#2209 sass#2210 sass#2211 sass#2212 sass#2213 sass#2214 sass#2215 #2216 sass#2217 sass#2218 sass#2219 sass#2220 sass#2221 sass#2222 sass#2223 sass#2224 sass#2225 sass#2226 sass#2227 sass#2228 sass#2229 sass#2230 sass#2231 sass#2232 sass#2233 sass#2234 sass#2235 sass#2236 sass#2237 sass#2238 sass#2239 sass#2240 sass#2241 sass#2242 sass#2243 sass#2244 sass#2245 sass#2246 sass#2247 sass#2248 sass#2249 sass#2250 sass#2251 sass#2252 sass#2253 sass#2254 sass#2255 sass#2256 sass#2257 sass#2258 sass#2259 sass#2260 sass#2261 sass#2262 sass#2263 sass#2264 sass#2265 sass#2266 sass#2267 sass#2268 sass#2269 sass#2270
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants