Skip to content

logging: implement WAL for net writer #7046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from
Draft

logging: implement WAL for net writer #7046

wants to merge 5 commits into from

Conversation

mohammed90
Copy link
Member

After wrestling with it for ~1.5 years, I caved and used Cody. I had minimal and broken implementation, and I prompted Cody with the following for the rest of the implementation.

The NetWriter logger writes log lines to the network synchronously, meaning it'll block the HTTP request handling if the network is slow. This is not ideal. The NetWriter should write to a WAL (Write-Ahead-Log) that is asynchrounously flushed to the network. When the server is stopping, it should flush the full WAL. When the server starts and WAL is present, it should append to it and try to flush it. The code currently is the start of an attempt to implement this but is incomplete and likely incorrect. Implement the requirement properly.

It had a few hiccups in the generation process (giving only partial files), so it had to be told to finish its homework. Then I asked it to generate the tests with this prompt:

can you add a test for the WAL implementation? The test should be in a new file named netwriter_test.go

Two of the test functions fail. Cody keeps trying to fix the tests instead of fixing the code. I also see unused struct field is introduced. Manual polishing work is WIP. Support is welcome.

Closes #5697

Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
@mohammed90 mohammed90 added feature ⚙️ New feature or request in progress 🏃‍♂️ Being actively worked on under review 🧐 Review is pending before merging optimization 📉 Performance or cost improvements labels Jun 2, 2025
Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
@mholt
Copy link
Member

mholt commented Jun 6, 2025

Wow 💦 Cool -- thanks for iterating on this. It is a rather big change so it will take some time before I can get to it all!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature ⚙️ New feature or request in progress 🏃‍♂️ Being actively worked on optimization 📉 Performance or cost improvements under review 🧐 Review is pending before merging
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement WAL writer for the net log writer
2 participants