-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Data race #36
Comments
I can't reproduce the bug, the above code runs without errors. What is your environment (os/go version/colly version)? |
colly version: d7069d1 (master) |
hmm.. I can reproduce if I add the |
@papa-stiflera you didn't paste the whole code you're using in order to be able to re-produce this. At the end of your logs: My output (win 10 x64, go 1.9.1); |
@kataras thanks for the debugging. My first assumption was the same, then I tried the above code, and the bug is reproducible - even with the below snippet: package main
import (
"github.com/asciimoo/colly"
)
func main() {
colly.NewCollector().Visit("https://en.wikipedia.org/")
} The strange thing is that if I change the URL to a service which doesn't support HTTP/2 (e.g. to UPDATE: |
@asciimoo It's funny because the data racer couldn't find that with the first try, I had to re-run the program more than 4 times to view the race log... Update: I managed to "fix" that by locking when |
@kataras unfortunately your suggested solution is not applicable, because it forbids parallelism in |
I know @asciimoo ...I suggested it as a temporary solution, I don't know the whole code base so I can't help any further for now, but if it's a net/http issue then you have to fill an issue there :/ |
The bug is reproducible without colly. package main
import (
"net/http"
"net/http/cookiejar"
)
func main() {
jar, _ := cookiejar.New(nil)
client := &http.Client{Jar: jar}
client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
lastRequest := via[len(via)-1]
req.Header = lastRequest.Header
return nil
}
client.Get("https://en.wikipedia.org/")
} Seems, the bug only appears if the client has a cookie jar and a custom redirect handler which writes to |
Example code:
Execution log:
The text was updated successfully, but these errors were encountered: