Concurrency issue in home-message-list (obj not instance of ByteArrayOutputStream) #2
Comments
|
I also tried adding a very naive locking with no success. Interestingly, it made things worse. Now many different errors show up: (def home-message-lock "")
(defn home-message-list [_]
(locking home-message-lock
(let [out (ByteArrayOutputStream. 4096)
writer (transit/writer out :json)]
(transit/write writer {:messages (db-get-messages)})
(.toString out))))
(let [x (->> (range 10000)
(pmap (fn [_]
(home-message-list nil)))
doall)]
nil)
;; => clojure.lang.ExceptionInfo: null object for public boolean java.io.File.exists()
;; => clojure.lang.ExceptionInfo: non-null object for public static java.lang.Object borkdude.graal.LockFix.lock(java.lang.Object,clojure.lang.IFn)
;; => clojure.lang.ExceptionInfo: null object for public synchronized java.lang.String java.io.ByteArrayOutputStream.toString()Strange. Perhaps some non-thread-safe/mutable context under the hood or something? But if so, where? httpkit, babashka? |
|
@brdloush Thank you for your very interesting and detailed analysis. I cannot answer directly where the problem is, but: This example project is an outflow of an old (year 2020, pre-scittle) project of mine. Not having made any analysis, it was clear to me that a richer back-end is desirable. So I made a fork called babashka-web, worked to get ring and reitit+muuntaja Graal compilable and made a back-end more in line with Luminus. My guess is that the result is thread-safe. |
|
I just tried running the server in regular clojure environment instead of babashka. My previous example with So it seems that the problem only occurs when executed via babashka. What I find insteresting is that when I use babashka, pass On the other hand, when I start the server with regular clojure, same Perhaps @borkdude might have some idea what might be the cause of this? 🙏 (btw I'm using |
|
btw @borkdude the same issue also happens in @holyjak's http server. https://gist.github.com/holyjak/36c6284c047ffb7573e8a34399de27d8. |
|
@brdloush In the Readme, I added a caveat that the code is not useful for production. |
|
@brdloush Can you make a very minimal repro? Then I can take a closer look. Preferably you would also make a GraalVM-native project with only httpkit and without sci, to see if this gives similar issues or not. Feel free to make an issue at the babashka repo for this. |
|
@brdloush On my laptop with this example: and |
|
Same with this guestbook actually, can't reproduce the issue. 0 request failed. |
|
With the file server I can reproduce the issue, I'm going to see if I can reproduce this somehow with a smaller example. |
|
I'm starting to think that |
|
The problem might be related to interop in general it looks like. |
|
It would be interesting to check if the problem also occurs with 0.4.4 since I made some changes related to interop in bb in 0.4.5. |
|
Minimal repro: Doing two interop on different kinds of objects in a function that is executed in parallel results in these kinds of errors. Very interesting. |
|
Here's my minimal case, but the one you just posted seems even more minimal :-) (require '[clojure.java.io :as io]
'[org.httpkit.server :as srv])
(import 'java.io.ByteArrayOutputStream)
(def port 8083)
(defn core-http-server []
(srv/run-server
(fn [_req]
{:headers {"Content-type" "application/transit+json"}
:body (let [out (ByteArrayOutputStream. 4096)]
(.exists (io/file "non-existing-file.txt"))
(.toString out)
"response body")
:status 200})
{:port port
:thread 8}))
(core-http-server)
@(promise)➜ simplehttp ab -c8 -n 10000 http://localhost:8083/messages 2>/dev/null | grep Failed
Failed requests: 54
➜ simplehttp ab -c8 -n 10000 http://localhost:8083/messages 2>/dev/null | grep Failed
Failed requests: 47
➜ simplehttp ab -c1 -n 10000 http://localhost:8083/messages 2>/dev/null | grep Failed
Failed requests: 0
➜ simplehttp ab -c1 -n 10000 http://localhost:8083/messages 2>/dev/null | grep Failed
Failed requests: 0
➜ simplehttp curl http://localhost:8083
response body% |
|
Btw we can maybe even remove the httpkit out of your minimal case, can't we? Something like |
|
Yes. More minimal repro: |
|
Tracking the issue here: babashka/babashka#884 |
|
When compiling babashka with GraalVM 21.1 instead if 21.0 it started working. Pfffff. @brdloush Would you try out this new binary? What's your OS? |
|
@borkdude Awesome 👍 🤞 Sure, I'll try. I'm on linux x64 ubuntu 21.04 |
|
I just compiled current babashka master |
|
Both enclosed versions are working fine 👏 What's more interesting is that it's also perfoming insanely well: ➜ babashka-scittle-guestbook git:(main) ab -c 8 -n 10000 http://localhost:8083/messages
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
Server Software: http-kit
Server Hostname: localhost
Server Port: 8083
Document Path: /messages
Document Length: 112 bytes
Concurrency Level: 8
Time taken for tests: 0.690 seconds
Complete requests: 10000
Failed requests: 0
Total transferred: 2470000 bytes
HTML transferred: 1120000 bytes
Requests per second: 14491.85 [#/sec] (mean)
Time per request: 0.552 [ms] (mean)
Time per request: 0.069 [ms] (mean, across all concurrent requests)
Transfer rate: 3495.59 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 0 0 0.1 0 2
Waiting: 0 0 0.1 0 2
Total: 0 1 0.1 1 2
Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 1
80% 1
90% 1
95% 1
98% 1
99% 1
100% 2 (longest request) |
|
@brdloush Excellent. I will release a new bb. |
|
I wonder what the real issue with that older GraalVM 21.0 was and which of the fixes in 21.1 actually fixed this specific problem. Hopefully they'll not re-introduce that issue in the future :-) |
|
@brdloush I have asked about this in the graal channel, I am also interested in finding this out. To make sure to prevent a regression, I am adding a test now. |
|
@brdloush @kloimhardt Released 0.4.6: https://github.com/babashka/babashka/blob/master/CHANGELOG.md#046 @kloimhardt I think you can remove the warning in the README now and change it to a recommendation to use 0.4.6. |
|
@brdloush Btw, thanks for discovering this! I'll be looking forward to discover which commit(s) fixed this in GraalVM but it looks like searching for a needle in a haystack. |
|
Impressive seeing the masters at work. |
Hi,
first let me say that this is a very interesting example project! Looking forward to see what else is possible with bb and scittle.
I tried running
abbenchmark to see what performance characteristics project like this can offer when it comes to API response rate and found out that it actually returns quite a lot of non-200 responses. They all seem to be caused byobject is not an instance of java.io.ByteArrayOutputStreamerror.Steps to reproduce:
clone the fresh copy of this repo
insert some sample message
There's following error written in server output
When I use
-c 1, the error doesn't show. With-c 2and greater, it does. So it may be caused by some non-thread-safe code.BTW I now also tried removing
messages.txtcompletely (and restarted the server) and the error is still there.There seems to be something non-thread-safe in
home-message-list. I tried following expressions in babashka nrepl sessionbb --nrepl-server:The text was updated successfully, but these errors were encountered: