Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace select(2) with poll(2) #1991

Merged
merged 2 commits into from Jan 11, 2021
Merged

Replace select(2) with poll(2) #1991

merged 2 commits into from Jan 11, 2021

Conversation

@rsmarples
Copy link
Contributor

@rsmarples rsmarples commented Jan 3, 2021

poll(2) is more modern, more efficient and a lot faster than
select(2). The trade-off is that we need to manage the
array of struct pollfd we send it a bit more intelligently
than just manipulating tables before each call to select(2).

This commit adds functions to dynamically resize the allocation
needed when new fd's need to be monitored or old ones discarded
and then sprinkles them where needed.

@rsmarples rsmarples force-pushed the rsmarples:poll branch 2 times, most recently from 6749989 to ede7e9d Jan 3, 2021
@allinurl
Copy link
Owner

@allinurl allinurl commented Jan 4, 2021

Thanks for submitting this. I'm still looking at the changes. Though, would you be able to run it through valgrind, so far I'm getting several Invalid reads on my end, wondering if you are seeing the same on your side? Thank you!

==24092== Memcheck, a memory error detector
==24092== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==24092== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==24092== Command: ./goaccess - -o /home/gerardoj/www/goaccess/a.html --real-time-html
==24092== Parent PID: 19780
==24092== 
==24092== Thread 2:
==24092== Invalid read of size 4
==24092==    at 0x165F69: ws_start (websocket.c:2759)
==24092==    by 0x13955C: start_server (gwsocket.c:335)
==24092==    by 0x4C57FA2: start_thread (pthread_create.c:486)
==24092==    by 0x4D6A4CE: clone (clone.S:95)
==24092==  Address 0x50d7fc0 is 32 bytes inside a block of size 40 free'd
==24092==    at 0x4837D7B: realloc (vg_replace_malloc.c:826)
==24092==    by 0x16670A: xrealloc (xmalloc.c:80)
==24092==    by 0x15FB5A: set_pollfd (websocket.c:199)
==24092==    by 0x1615F1: accept_client (websocket.c:1037)
==24092==    by 0x1640CB: handle_accept (websocket.c:2194)
==24092==    by 0x166051: ws_start (websocket.c:2776)
==24092==    by 0x13955C: start_server (gwsocket.c:335)
==24092==    by 0x4C57FA2: start_thread (pthread_create.c:486)
==24092==    by 0x4D6A4CE: clone (clone.S:95)
==24092==  Block was alloc'd at
==24092==    at 0x4837D7B: realloc (vg_replace_malloc.c:826)
==24092==    by 0x16670A: xrealloc (xmalloc.c:80)
==24092==    by 0x15FB5A: set_pollfd (websocket.c:199)
==24092==    by 0x1615F1: accept_client (websocket.c:1037)
==24092==    by 0x1640CB: handle_accept (websocket.c:2194)
==24092==    by 0x166051: ws_start (websocket.c:2776)
==24092==    by 0x13955C: start_server (gwsocket.c:335)
==24092==    by 0x4C57FA2: start_thread (pthread_create.c:486)
==24092==    by 0x4D6A4CE: clone (clone.S:95)
==24092== 
==24092== Invalid read of size 4
==24092==    at 0x165FA8: ws_start (websocket.c:2765)
==24092==    by 0x13955C: start_server (gwsocket.c:335)
==24092==    by 0x4C57FA2: start_thread (pthread_create.c:486)
==24092==    by 0x4D6A4CE: clone (clone.S:95)
==24092==  Address 0x50d7fc0 is 32 bytes inside a block of size 40 free'd
==24092==    at 0x4837D7B: realloc (vg_replace_malloc.c:826)
==24092==    by 0x16670A: xrealloc (xmalloc.c:80)
==24092==    by 0x15FB5A: set_pollfd (websocket.c:199)
==24092==    by 0x1615F1: accept_client (websocket.c:1037)
==24092==    by 0x1640CB: handle_accept (websocket.c:2194)
==24092==    by 0x166051: ws_start (websocket.c:2776)
==24092==    by 0x13955C: start_server (gwsocket.c:335)
==24092==    by 0x4C57FA2: start_thread (pthread_create.c:486)
==24092==    by 0x4D6A4CE: clone (clone.S:95)
==24092==  Block was alloc'd at
==24092==    at 0x4837D7B: realloc (vg_replace_malloc.c:826)
==24092==    by 0x16670A: xrealloc (xmalloc.c:80)
==24092==    by 0x15FB5A: set_pollfd (websocket.c:199)
==24092==    by 0x1615F1: accept_client (websocket.c:1037)
==24092==    by 0x1640CB: handle_accept (websocket.c:2194)
==24092==    by 0x166051: ws_start (websocket.c:2776)
==24092==    by 0x13955C: start_server (gwsocket.c:335)
==24092==    by 0x4C57FA2: start_thread (pthread_create.c:486)
==24092==    by 0x4D6A4CE: clone (clone.S:95)
==24092== 
==24092== Invalid read of size 4
==24092==    at 0x165FE1: ws_start (websocket.c:2769)
==24092==    by 0x13955C: start_server (gwsocket.c:335)
==24092==    by 0x4C57FA2: start_thread (pthread_create.c:486)
==24092==    by 0x4D6A4CE: clone (clone.S:95)
==24092==  Address 0x50d7fc0 is 32 bytes inside a block of size 40 free'd
==24092==    at 0x4837D7B: realloc (vg_replace_malloc.c:826)
==24092==    by 0x16670A: xrealloc (xmalloc.c:80)
==24092==    by 0x15FB5A: set_pollfd (websocket.c:199)
==24092==    by 0x1615F1: accept_client (websocket.c:1037)
==24092==    by 0x1640CB: handle_accept (websocket.c:2194)
==24092==    by 0x166051: ws_start (websocket.c:2776)
==24092==    by 0x13955C: start_server (gwsocket.c:335)
==24092==    by 0x4C57FA2: start_thread (pthread_create.c:486)
==24092==    by 0x4D6A4CE: clone (clone.S:95)
==24092==  Block was alloc'd at
==24092==    at 0x4837D7B: realloc (vg_replace_malloc.c:826)
==24092==    by 0x16670A: xrealloc (xmalloc.c:80)
==24092==    by 0x15FB5A: set_pollfd (websocket.c:199)
==24092==    by 0x1615F1: accept_client (websocket.c:1037)
==24092==    by 0x1640CB: handle_accept (websocket.c:2194)
==24092==    by 0x166051: ws_start (websocket.c:2776)
==24092==    by 0x13955C: start_server (gwsocket.c:335)
==24092==    by 0x4C57FA2: start_thread (pthread_create.c:486)
==24092==    by 0x4D6A4CE: clone (clone.S:95)
==24092== 
==24092== Invalid read of size 4
==24092==    at 0x166028: ws_start (websocket.c:2773)
==24092==    by 0x13955C: start_server (gwsocket.c:335)
==24092==    by 0x4C57FA2: start_thread (pthread_create.c:486)
==24092==    by 0x4D6A4CE: clone (clone.S:95)
==24092==  Address 0x50d7fc0 is 32 bytes inside a block of size 40 free'd
==24092==    at 0x4837D7B: realloc (vg_replace_malloc.c:826)
==24092==    by 0x16670A: xrealloc (xmalloc.c:80)
==24092==    by 0x15FB5A: set_pollfd (websocket.c:199)
==24092==    by 0x1615F1: accept_client (websocket.c:1037)
==24092==    by 0x1640CB: handle_accept (websocket.c:2194)
==24092==    by 0x166051: ws_start (websocket.c:2776)
==24092==    by 0x13955C: start_server (gwsocket.c:335)
==24092==    by 0x4C57FA2: start_thread (pthread_create.c:486)
==24092==    by 0x4D6A4CE: clone (clone.S:95)
==24092==  Block was alloc'd at
==24092==    at 0x4837D7B: realloc (vg_replace_malloc.c:826)
==24092==    by 0x16670A: xrealloc (xmalloc.c:80)
==24092==    by 0x15FB5A: set_pollfd (websocket.c:199)
==24092==    by 0x1615F1: accept_client (websocket.c:1037)
==24092==    by 0x1640CB: handle_accept (websocket.c:2194)
==24092==    by 0x166051: ws_start (websocket.c:2776)
==24092==    by 0x13955C: start_server (gwsocket.c:335)
==24092==    by 0x4C57FA2: start_thread (pthread_create.c:486)
==24092==    by 0x4D6A4CE: clone (clone.S:95)
==24092== 
==24092== Invalid read of size 2
==24092==    at 0x166058: ws_start (websocket.c:2779)
==24092==    by 0x13955C: start_server (gwsocket.c:335)
==24092==    by 0x4C57FA2: start_thread (pthread_create.c:486)
==24092==    by 0x4D6A4CE: clone (clone.S:95)
==24092==  Address 0x50d7fc6 is 38 bytes inside a block of size 40 free'd
==24092==    at 0x4837D7B: realloc (vg_replace_malloc.c:826)
==24092==    by 0x16670A: xrealloc (xmalloc.c:80)
==24092==    by 0x15FB5A: set_pollfd (websocket.c:199)
==24092==    by 0x1615F1: accept_client (websocket.c:1037)
==24092==    by 0x1640CB: handle_accept (websocket.c:2194)
==24092==    by 0x166051: ws_start (websocket.c:2776)
==24092==    by 0x13955C: start_server (gwsocket.c:335)
==24092==    by 0x4C57FA2: start_thread (pthread_create.c:486)
==24092==    by 0x4D6A4CE: clone (clone.S:95)
==24092==  Block was alloc'd at
==24092==    at 0x4837D7B: realloc (vg_replace_malloc.c:826)
==24092==    by 0x16670A: xrealloc (xmalloc.c:80)
==24092==    by 0x15FB5A: set_pollfd (websocket.c:199)
==24092==    by 0x1615F1: accept_client (websocket.c:1037)
==24092==    by 0x1640CB: handle_accept (websocket.c:2194)
==24092==    by 0x166051: ws_start (websocket.c:2776)
==24092==    by 0x13955C: start_server (gwsocket.c:335)
==24092==    by 0x4C57FA2: start_thread (pthread_create.c:486)
==24092==    by 0x4D6A4CE: clone (clone.S:95)
==24092== 
==24092== Invalid read of size 2
==24092==    at 0x16609D: ws_start (websocket.c:2786)
==24092==    by 0x13955C: start_server (gwsocket.c:335)
==24092==    by 0x4C57FA2: start_thread (pthread_create.c:486)
==24092==    by 0x4D6A4CE: clone (clone.S:95)
==24092==  Address 0x50d7fc6 is 38 bytes inside a block of size 40 free'd
==24092==    at 0x4837D7B: realloc (vg_replace_malloc.c:826)
==24092==    by 0x16670A: xrealloc (xmalloc.c:80)
==24092==    by 0x15FB5A: set_pollfd (websocket.c:199)
==24092==    by 0x1615F1: accept_client (websocket.c:1037)
==24092==    by 0x1640CB: handle_accept (websocket.c:2194)
==24092==    by 0x166051: ws_start (websocket.c:2776)
==24092==    by 0x13955C: start_server (gwsocket.c:335)
==24092==    by 0x4C57FA2: start_thread (pthread_create.c:486)
==24092==    by 0x4D6A4CE: clone (clone.S:95)
==24092==  Block was alloc'd at
==24092==    at 0x4837D7B: realloc (vg_replace_malloc.c:826)
==24092==    by 0x16670A: xrealloc (xmalloc.c:80)
==24092==    by 0x15FB5A: set_pollfd (websocket.c:199)
==24092==    by 0x1615F1: accept_client (websocket.c:1037)
==24092==    by 0x1640CB: handle_accept (websocket.c:2194)
==24092==    by 0x166051: ws_start (websocket.c:2776)
==24092==    by 0x13955C: start_server (gwsocket.c:335)
==24092==    by 0x4C57FA2: start_thread (pthread_create.c:486)
==24092==    by 0x4D6A4CE: clone (clone.S:95)
==24092== 
==24092== 
==24092== HEAP SUMMARY:
==24092==     in use at exit: 8 bytes in 1 blocks
==24092==   total heap usage: 51,324 allocs, 51,323 frees, 21,539,333 bytes allocated
==24092== 
==24092== LEAK SUMMARY:
==24092==    definitely lost: 0 bytes in 0 blocks
==24092==    indirectly lost: 0 bytes in 0 blocks
==24092==      possibly lost: 0 bytes in 0 blocks
==24092==    still reachable: 8 bytes in 1 blocks
==24092==         suppressed: 0 bytes in 0 blocks
==24092== Reachable blocks (those to which a pointer was found) are not shown.
==24092== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==24092== 
==24092== For counts of detected and suppressed errors, rerun with: -v
==24092== ERROR SUMMARY: 75 errors from 6 contexts (suppressed: 0 from 0)
@rsmarples
Copy link
Contributor Author

@rsmarples rsmarples commented Jan 5, 2021

I'll find some time for this.
We don't have valgrind on NetBSD, but we do have the LLVM address sanitisers in our toolchain, it's just problematic when the host box lacks the compiler.

@allinurl
Copy link
Owner

@allinurl allinurl commented Jan 5, 2021

I'll take a look at it as well and see what I find. Thanks again!

@rsmarples
Copy link
Contributor Author

@rsmarples rsmarples commented Jan 6, 2021

PR #1993 is required as the first stepping stone to get address sanitisation to work via the clang compiler.
Another PR will follow to fix a issue compiling it specific to NetBSD.

@rsmarples rsmarples force-pushed the rsmarples:poll branch from ede7e9d to 7eeb2c0 Jan 7, 2021
poll(2) is more modern, more efficient and a lot faster than
select(2). The trade-off is that we need to manage the
array of struct pollfd we send it a bit more intelligently
than just manipulating tables before each call to select(2).

This commit adds functions to dynamically resize the allocation
needed when new fd's need to be monitored or old ones discarded
and then sprinkles them where needed.
@rsmarples rsmarples force-pushed the rsmarples:poll branch from 7eeb2c0 to 051e14b Jan 7, 2021
@rsmarples
Copy link
Contributor Author

@rsmarples rsmarples commented Jan 7, 2021

Should be fixed now.

@allinurl allinurl merged commit b85fc6b into allinurl:master Jan 11, 2021
1 check passed
1 check passed
test
Details
@allinurl
Copy link
Owner

@allinurl allinurl commented Jan 11, 2021

Merged. Thanks again for fixing this and submitting this PR. I added a fix (98d6947) for a leak on the last fdstate. Let me know if it makes sense adding that free call at point.

@rsmarples
Copy link
Contributor Author

@rsmarples rsmarples commented Jan 11, 2021

Merged. Thanks again for fixing this and submitting this PR. I added a fix (98d6947) for a leak on the last fdstate. Let me know if it makes sense adding that free call at point.

So that means that an fd being monitored wasn't closed.
If all fd's being monitored are closed then fdstate is already freed here

Working out which fd isn't being closed and ensuring it is would be the cleaner approach if you feel that's important.
Your's at least is the safety net.

If you ever have plans to open anothe websocket afterwards, you might want to set vars to NULL after freeing them, especially static ones such as fdstate.
Otherwise I'm fine with it :)

@allinurl
Copy link
Owner

@allinurl allinurl commented Jan 11, 2021

I see what you mean, yeah it's definitely not a biggie as it appears it was only upon stopping the server, though, mostly so it doesn't raise any flags on my tests :) I just added a minor commit to NULL in case we ever need to restart it.

@rsmarples rsmarples deleted the rsmarples:poll branch Jan 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants