gor traffic copy write file error, partly out of order

[wjyGithub]

As shown above, Access-Token contains the next request line information

[buger]

Hi!

What version of GoReplay do you use?

Is this output from a recorded file?

Can you show a command which you use to record this traffic?

And finally if it is easy to replicate, any chance you can record small portion of traffic and send me to support@goreplay.org?

Thank you!

[wjyGithub]

gor version is 1.2.0

Command executed：sudo ./gor --input-raw :80 --output-file ./gor.log --exit-after 600s --output-file-append --http-allow-header Host:xxx.xxx.com --http-allow-url /v1/xxx/unread_count.json -http-disallow-url /v1/xxx/detail.json --http-disallow-url /v1/xxx/ban_list.json --http-allow-method GET

Currently we are using it online, so it involves security issues, it may not be convenient to give the copied files。sorry

[buger]

I would recommend trying 1.3, this code has changed a lot

[wjyGithub]

gor version : 1.3.0
Operating environment: run gor commond in Docker
Command executed: sudo ./gor --input-raw :80 --output-file /wow/gor.log --exit-after 1800000ms --output-file-append --http-allow-header Host:xxx.xxx.com --http-allow-url /v1/xxx/unread_count.json --http-allow-method GET --http-set-header isStressTest:1

As shown above, using gor1.3.0 to copy traffic, it is found that the problem is more serious. There are a large number of missing request lines.

I guess that there may be a concurrency problem in the process of writing files.

[buger]

Indeed it is very weird! I wonder if it happen same if you remove http-set and http-allow filters... Just want try to isolate the issue.

--

Sincerely yours, Leonid Bugaev https://goreplay.org - test your system with real data @buger <https://twitter.com/buger> - me on twitter

[buger]

@wjyGithub looks bad indeed! Can you try latest 1.3.3 pls?

[wjyGithub]

About this question，I may need to add background。

The scenario where this problem occurs is usually when the QPS is relatively high.

Our online QPS is usually more than 10,000.

At the same time, the online connection is continuously maintained。

I want to reproduce this problem locally, but I cannot simulate a higher QPS.

I have a guess here, is it because the Http connection is KeepAlive when the gopacket library captures the message, it can’t distinguish the front and back requests correctly, causing the previous request to be overwritten by the next request

[wjyGithub]

Similar to the tcpdump tool, you can set a buffer size

[torysuper]

Similar to the tcpdump tool, you can set a buffer size

I have the same situation with you.
There are 2/3 request unrecard.
can i ask you how to solve this problem.