EVFILT_USER is not available on NetBSD

[travispaul]

NetBSD's kqueue does not support EVFILT_USER and so NNG does not currently build on that platform.

As a quick hack I was able to #ifdef around the locations where EVFILT_USER is used and the project was able to build. So the lack of EVFILT_USER appears to be the only build issue currently.

NNG & Platform details.

nng v1.3.0
NetBSD 9.0 amd64

Expected Behavior

NNG builds on NetBSD

Actual Behavior

[ 38%] Building C object src/CMakeFiles/nng.dir/platform/posix/posix_pollq_kqueue.c.o
In file included from /usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c:19:0:
/usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c: In function 'nni_posix_pfd_fini':
/usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c:135:14: error: 'EVFILT_USER' undeclared (first use in this function); did you mean 'EVFILT_TIMER'?
      &ev, 0, EVFILT_USER, EV_ENABLE | EV_CLEAR, NOTE_TRIGGER, 0, NULL);
              ^
/usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c:135:14: note: each undeclared identifier is reported only once for each function it appears in
/usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c:135:49: error: 'NOTE_TRIGGER' undeclared (first use in this function); did you mean 'NOTE_WRITE'?
      &ev, 0, EVFILT_USER, EV_ENABLE | EV_CLEAR, NOTE_TRIGGER, 0, NULL);
                                                 ^
[ 38%] Building C object src/CMakeFiles/nng_testlib.dir/platform/posix/posix_udp.c.o
/usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c: In function 'nni_posix_poll_thr':
/usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c:251:9: error: 'EVFILT_USER' undeclared (first use in this function); did you mean 'EVFILT_TIMER'?
    case EVFILT_USER:
         ^~~~~~~~~~~
         EVFILT_TIMER
In file included from /usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c:19:0:
/usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c: In function 'nni_posix_pollq_add_wake_evt':
/usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c:297:17: error: 'EVFILT_USER' undeclared (first use in this function); did you mean 'EVFILT_TIMER'?
  EV_SET(&ev, 0, EVFILT_USER, EV_ADD | EV_CLEAR, 0, 0, NULL);
                 ^
--- src/CMakeFiles/nng.dir/platform/posix/posix_pollq_kqueue.c.o ---
*** [src/CMakeFiles/nng.dir/platform/posix/posix_pollq_kqueue.c.o] Error code 1

make[2]: stopped in /usr/pkgsrc/wip/nng/work/nng-1.3.0
[ 38%] Building C object src/CMakeFiles/nng_testlib.dir/platform/posix/posix_pollq_kqueue.c.o
1 error

make[2]: stopped in /usr/pkgsrc/wip/nng/work/nng-1.3.0
--- src/CMakeFiles/nng.dir/all ---
*** [src/CMakeFiles/nng.dir/all] Error code 2

make[1]: stopped in /usr/pkgsrc/wip/nng/work/nng-1.3.0
In file included from /usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c:19:0:
/usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c: In function 'nni_posix_pfd_fini':
/usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c:135:14: error: 'EVFILT_USER' undeclared (first use in this function); did you mean 'EVFILT_TIMER'?
      &ev, 0, EVFILT_USER, EV_ENABLE | EV_CLEAR, NOTE_TRIGGER, 0, NULL);

/usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c:135:14: note: each undeclared identifier is reported only once for each function it appears in
/usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c:135:49: error: 'NOTE_TRIGGER' undeclared (first use in this function); did you mean 'NOTE_WRITE'?
      &ev, 0, EVFILT_USER, EV_ENABLE | EV_CLEAR, NOTE_TRIGGER, 0, NULL);
                                                 ^
/usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c: In function 'nni_posix_poll_thr':
/usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c:251:9: error: 'EVFILT_USER' undeclared (first use in this function); did you mean 'EVFILT_TIMER'?
    case EVFILT_USER:
         ^~~~~~~~~~~
         EVFILT_TIMER
In file included from /usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c:19:0:
/usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c: In function 'nni_posix_pollq_add_wake_evt':
/usr/pkgsrc/wip/nng/work/nng-1.3.0/src/platform/posix/posix_pollq_kqueue.c:297:17: error: 'EVFILT_USER' undeclared (first use in this function); did you mean 'EVFILT_TIMER'?
  EV_SET(&ev, 0, EVFILT_USER, EV_ADD | EV_CLEAR, 0, 0, NULL);
                 ^
--- src/CMakeFiles/nng_testlib.dir/platform/posix/posix_pollq_kqueue.c.o ---
*** [src/CMakeFiles/nng_testlib.dir/platform/posix/posix_pollq_kqueue.c.o] Error code 1

Steps to Reproduce

Build on NetBSD 9.0.

[gdamore]

I suspect you may not be able to use the kqueue() infrastructure -- and may have to resort to the poll() framework. The issue is that we rely on the EVFILT_USER to wake up kqueue when changing the list of descriptors to wake up, and to serialize access to the kqueue handling thread. (For example, closing the descriptor will prevent new callbacks form occurring, but we have no way to serialize against callbacks which may be running on that thread.)

We could design a workaround for this. But I'm not sure how much I want to invest in that. Honestly NetBSD should probably just implement EVFILT_USER at this point.

I may conditionalize the support for kqueue on having EVFILT_USER just to permit things to go forward.

[gdamore]

I've got a PR that basically prevents the use of kqueue unless you have EVFILT_USER. Without a working EVFILT_USER we would have to use cross signaling (e.g. via socketpair descriptors) which is going to impair scalability anyway. So you'll have a functional library, but you won't be as performant or scalable as FreeBSD on either NetBSD or OpenBSD. Someone in those projects should look at lifting the code from FreeBSD to implement EVFILT_USER. It can bring real significant wins in performance and scalability.

[gdamore]

If someone who is using NetBSD or OpenBSD wanted to contribute a working socketpair() based replacement (perhaps borrowing the logic from the poll() implementation) I would be open to that. I don't have the time to do this myself, nor the inclination, nor the platform(s) to test and validate such changes myself.