Concurrency issues

[andyleap]

Something is making the application return 404's randomly when refreshing too fast. The errors happen all over the place, even at the home url "/". The issue seems to be that something can't handle concurrent access, as when I wrap the macaron http handler in a func like

var mutex sync.Mutex

handler := func (w http.ResponseWriter, req *http.Request) {
        mutex.Lock()
        m.ServeHTTP(w, req)
        mutex.Unlock()
}

and pass that like http.ListenAndServe(listenAddr, http.HandlerFunc(handler)), everything works fine.

Rather serious issue if the system can't handle multiple requests at once.....

[Unknwon]

Thanks your feedback!

But I think the change you made just make macaron can only handle 1 request at a time? Not sure.

I've did ab test on 1000 connections and 10000 requests and didn't seem to have this random situation, so can you provide more information?

[Unknwon]

Thanks your feedback!

But I think the change you made just make macaron can only handle 1 request at a time? Not sure.

I've did ab test on 1000 connections and 10000 requests and didn't seem to have this random situation, so can you provide more information?

[andyleap]

I just put compiled a -race build and there are data races ALL OVER THE PLACE. There's some seriously bad stuff going on here.

[andyleap]

I just put compiled a -race build and there are data races ALL OVER THE PLACE. There's some seriously bad stuff going on here.

[Unknwon]

Good point, haven't tried -race flag before. BTW, is this should be considered as Gogs issue or Macaron?

[Unknwon]

Good point, haven't tried -race flag before. BTW, is this should be considered as Gogs issue or Macaron?

[andyleap]

There's data races in the cron, in the orm, in the captcha, and that's just what I've seen in 5 minutes. I'd say it's an everywhere issue...

[andyleap]

There's data races in the cron, in the orm, in the captcha, and that's just what I've seen in 5 minutes. I'd say it's an everywhere issue...

[Unknwon]

In my point of view, as long as no deadlock..data race is unavoidable.

[Unknwon]

In my point of view, as long as no deadlock..data race is unavoidable.

[andyleap]

I really don't mean to be rude, but they are hardly unavoidable, and while not causing deadlocks, they are causing issues. I'm not sure I can use this software as it is.

[andyleap]

I really don't mean to be rude, but they are hardly unavoidable, and while not causing deadlocks, they are causing issues. I'm not sure I can use this software as it is.

[Unknwon]

If to use things like sync.Mutex, I think there is really no concurrent at all, just a single thread..

[Unknwon]

If to use things like sync.Mutex, I think there is really no concurrent at all, just a single thread..

[andyleap]

and if I don't, I get 404's on explore, dashboard, user profiles, and more. Think I'll take single thread over that.

[andyleap]

and if I don't, I get 404's on explore, dashboard, user profiles, and more. Think I'll take single thread over that.

[Unknwon]

What is your system and version of Gogs? And is it possible to reproduce on https://try.gogs.io?

[Unknwon]

What is your system and version of Gogs? And is it possible to reproduce on https://try.gogs.io?

[andyleap]

0.5.7.1105 Beta on linux. Tried ab against it compiled without the mutex, and gogs crashed.

[andyleap]

0.5.7.1105 Beta on linux. Tried ab against it compiled without the mutex, and gogs crashed.

[andyleap]

With roughly 99,950 or so lines of stack traces and an error of too many open files.

[andyleap]

With roughly 99,950 or so lines of stack traces and an error of too many open files.

[Unknwon]

0.5.7.1105 Beta on linux. Tried ab against it compiled without the mutex, and gogs crashed.

What is the behave of crashed of Gogs?

With roughly 99,950 or so lines of stack traces and an error of too many open files.

This is limited by your ulimit -n.

[Unknwon]

0.5.7.1105 Beta on linux. Tried ab against it compiled without the mutex, and gogs crashed.

What is the behave of crashed of Gogs?

With roughly 99,950 or so lines of stack traces and an error of too many open files.

This is limited by your ulimit -n.

[lifeofguenter]

Is this maybe the reason why sometimes the avatar won't load?

-> I have avatar caching activated
-> on a site where a lot of avatars are being loaded (logged in -> dashboard)
-> avatar cache is being refreshed

Result:
-> some avatars (or all) do not load.

Also some improvements I would make with the avatar caching:

1. ability to configure the TTL, per default it can be at least a day (though thinking that even 7-30 days is ok)
2. add an expire header with the same value of the TTL, so client does not need to download the avatar each time
3. fix the ETAG (currently it is "Etag:size(80)"

[lifeofguenter]

Is this maybe the reason why sometimes the avatar won't load?

-> I have avatar caching activated
-> on a site where a lot of avatars are being loaded (logged in -> dashboard)
-> avatar cache is being refreshed

Result:
-> some avatars (or all) do not load.

Also some improvements I would make with the avatar caching:

1. ability to configure the TTL, per default it can be at least a day (though thinking that even 7-30 days is ok)
2. add an expire header with the same value of the TTL, so client does not need to download the avatar each time
3. fix the ETAG (currently it is "Etag:size(80)"

[Unknwon]

@lifeofguenter thanks your info! I'll take a look ASAP.

[Unknwon]

@lifeofguenter thanks your info! I'll take a look ASAP.

[andyleap]

    With roughly 99,950 or so lines of stack traces and an error of too many open files.

This is limited by your ulimit -n.

Simple load test shouldn't kill the server due to open file limits, that's insane.

Is this maybe the reason why sometimes the avatar won't load?

-> I have avatar caching activated
-> on a site where a lot of avatars are being loaded (logged in -> dashboard)
-> avatar cache is being refreshed

Result:
-> some avatars (or all) do not load.

Yeah, I was having this issue to, tracked it down to the server randomly throwing 404's on the avatar urls. Disabling concurrency fixed it right away.

[andyleap]

    With roughly 99,950 or so lines of stack traces and an error of too many open files.

This is limited by your ulimit -n.

Simple load test shouldn't kill the server due to open file limits, that's insane.

Is this maybe the reason why sometimes the avatar won't load?

-> I have avatar caching activated
-> on a site where a lot of avatars are being loaded (logged in -> dashboard)
-> avatar cache is being refreshed

Result:
-> some avatars (or all) do not load.

Yeah, I was having this issue to, tracked it down to the server randomly throwing 404's on the avatar urls. Disabling concurrency fixed it right away.

[Unknwon]

@andyleap thanks for the update!

[Unknwon]

@andyleap thanks for the update!

[Unknwon]

The primary race problem should have been fixed by Macaron itself. So close this now.

[Unknwon]

The primary race problem should have been fixed by Macaron itself. So close this now.