Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenBSD support #649

Open
douglas-carmichael opened this issue Oct 24, 2019 · 25 comments

Comments

@douglas-carmichael
Copy link

@douglas-carmichael douglas-carmichael commented Oct 24, 2019

I was able to patch Zeek 3.0.0 to compile on OpenBSD 6.6 with the attached patches.
zeek-3.0.0-openbsd-Source.cc.patch.txt

@timwoj

This comment has been minimized.

Copy link
Contributor

@timwoj timwoj commented Oct 24, 2019

Would you be willing to turn this into a pull request? @rsmmr and I were just talking this morning about whether OpenBSD works or not, and then this comes out of the blue. CAF is a git submodule, so you'll need to create a pull request over there too (https://github.com/actor-framework/actor-framework/).

@douglas-carmichael

This comment has been minimized.

Copy link
Author

@douglas-carmichael douglas-carmichael commented Oct 25, 2019

I just created pull request #653 (comparing the master and 3.0 branches) but GitHub seems to have picked up more commits than I wanted.

@douglas-carmichael

This comment has been minimized.

Copy link
Author

@douglas-carmichael douglas-carmichael commented Oct 25, 2019

After looking at the actor framework, I can't see what branches to compare to commit my pull request.

However, I did create pull request #949 (https://github.com/actor-framework/actor-framework/pull/949) which includes my patch set.

@timwoj

This comment has been minimized.

Copy link
Contributor

@timwoj timwoj commented Oct 25, 2019

The problem with your PR is that you requested a merge from release/3.0 to master on the main git repo and I don't see your commit in the list of commits on the PR.

Did you fork a copy of the main repo and make the changes on a branch in that fork? If so, you need to make sure that the PR goes from that fork's branch to the main repo.

It looks like the same issue with your PR on the CAF side too.

@jsiwek

This comment has been minimized.

Copy link
Member

@jsiwek jsiwek commented Oct 25, 2019

I closed the Zeek PR, please create a fork/branch and request merging into the master branch.

If the aim is to get the changes into Zeek 3.0.1, that's a bit lengthier process. First, we need to review the changes. Once they're approved, they get merged into master and we can consider backporting them into a patch release (e.g. 3.0.1), but since this also depends on patching CAF (actor-framework), we'd also need a 0.16.5 release of CAF which includes the other OpenBSD-support patches (@Neverlord could give more an opinion on plausibility of that).

@douglas-carmichael

This comment has been minimized.

Copy link
Author

@douglas-carmichael douglas-carmichael commented Oct 25, 2019

How would I go about forking a copy of both repositories? Would I do it from within the GitHub interface?
(I was working from a downloaded tarball, not a git clone.)

@timwoj

This comment has been minimized.

Copy link
Contributor

@timwoj timwoj commented Oct 25, 2019

Ah hah! That definitely explains it.

From the main page for the Zeek repo, press the Fork button at the top right. If you get a popup to select your account, select and then github should start forking it to http://github.com/douglas-carmichael/zeek. Once that's done, clone the repo and do the work on that clone. Once you've pushed your change to your fork, you can open a pull request from your fork into the main zeek repo.

We've been talking about writing a blog post recently about how to contribute external changes to zeek, especially for people new to git and github. I guess this shows it'd be a good thing get done soon. 😃

@douglas-carmichael

This comment has been minimized.

Copy link
Author

@douglas-carmichael douglas-carmichael commented Oct 25, 2019

Thanks! I'll be eagerly awaiting said guide.
Would any of you be able to delete my previous PRs then?

@douglas-carmichael

This comment has been minimized.

Copy link
Author

@douglas-carmichael douglas-carmichael commented Oct 25, 2019

I just committed two "proper" PRs from my cloned repositories (#656 on the zeek side and #950 on the CAF side.)

Here's the URL to the CAF PR: actor-framework/actor-framework#950.
Hope these are acceptable!

@timwoj

This comment has been minimized.

Copy link
Contributor

@timwoj timwoj commented Oct 25, 2019

Thanks, those look much better.

Like @jsiwek mentioned CAF is an external project with their own release cycle. We'll have to wait for @Neverlord to cut a new release including it before it gets into a Zeek release.

@jsiwek jsiwek referenced this issue Oct 25, 2019
15 of 18 tasks complete
@Neverlord

This comment has been minimized.

Copy link
Member

@Neverlord Neverlord commented Oct 27, 2019

we'd also need a 0.16.5 release of CAF which includes the other OpenBSD-support patches

I see no blockers for backporting (cherry-picking, really) this patch for a new 0.16 tag.

@jsiwek jsiwek added this to Unassigned / Todo in Release 3.1.0 via automation Oct 30, 2019
@jsiwek jsiwek added this to the 3.1.0 milestone Oct 30, 2019
jsiwek added a commit that referenced this issue Oct 30, 2019
@jsiwek

This comment has been minimized.

Copy link
Member

@jsiwek jsiwek commented Oct 30, 2019

The Zeek-side of OpenBSD support are these patches now in master:

49c59ce
zeek/cmake@e8ddcae

I'll leave this issue open to remind about needing CAF updates to fully fix the OpenBSD build:

Zeek master and 3.1.0 would need a CAF 0.17.x.
Zeek 3.0.1 would need a CAF 0.16.x.

@douglas-carmichael

This comment has been minimized.

Copy link
Author

@douglas-carmichael douglas-carmichael commented Nov 5, 2019

I was finally able to get zeek from git master to configure properly by putting CAF in /usr/local, but even without --with-caf it still refuses to build because the in-built CAF doesn't support OpenBSD:


[ 7%] Building CXX object libcaf_core/CMakeFiles/libcaf_core_shared.dir/sec_strings.cpp.o
In file included from /home/dcarmich/zeek-git/zeek/build/aux/broker/caf-build/libcaf_core/sec_strings.cpp:1:
In file included from /home/dcarmich/zeek-git/zeek/build/aux/broker/caf-build/libcaf_core/caf/sec.hpp:26:
In file included from /home/dcarmich/zeek-git/zeek/aux/broker/3rdparty/caf/libcaf_core/caf/make_message.hpp:25:
In file included from /home/dcarmich/zeek-git/zeek/aux/broker/3rdparty/caf/libcaf_core/caf/message.hpp:26:
/home/dcarmich/zeek-git/zeek/aux/broker/3rdparty/caf/libcaf_core/caf/config.hpp:217:4: error:
Platform and/or compiler not supported

error Platform and/or compiler not supported

^
1 error generated.
*** Error 1 in aux/broker/caf-build (libcaf_core/CMakeFiles/libcaf_core_shared.dir/build.make:150 'libcaf_core/CMakeFiles/libcaf_core_shared.dir/sec_strings.cpp.o')
*** Error 1 in aux/broker/caf-build (CMakeFiles/Makefile2:245 'libcaf_core/CMakeFiles/libcaf_core_shared.dir/all')
*** Error 1 in aux/broker/caf-build (Makefile:130 'all')
*** Error 1 in . (aux/broker/CMakeFiles/project_caf.dir/build.make:118 'aux/broker/caf-ep/src/project_caf-stamp/project_caf-project_caf_build_step')
*** Error 1 in . (CMakeFiles/Makefile2:399 'aux/broker/CMakeFiles/project_caf.dir/all')
*** Error 1 in /home/dcarmich/zeek-git/zeek/build (Makefile:152 'all')


@jsiwek

This comment has been minimized.

Copy link
Member

@jsiwek jsiwek commented Nov 5, 2019

@douglas-carmichael the embedded CAF that Zeek uses hasn't been updated yet and we usually don't do that until there's a tagged release, so please manually install CAF and/or use --with-caf in the meantime.

@douglas-carmichael

This comment has been minimized.

Copy link
Author

@douglas-carmichael douglas-carmichael commented Nov 6, 2019

I used --with-caf and still had that issue despite CAF being installed in /usr/local.

@douglas-carmichael

This comment has been minimized.

Copy link
Author

@douglas-carmichael douglas-carmichael commented Nov 6, 2019

After I manually installed CAF from the topic/openbsd branch into the Zeek source tree, I was able to get as far as linking the zeek executable when ld complained about not finding libbinpac:

[ 98%] Linking CXX executable zeek
ld: error: unable to find library -llibbinpac.so.0.54-7
c++: error: linker command failed with exit code 1 (use -v to see invocation)

zeek-build-openbsd-11062019.txt

@jsiwek jsiwek self-assigned this Nov 6, 2019
jsiwek added a commit to zeek/binpac that referenced this issue Nov 7, 2019
OpenBSD shared library names are like "libfoo.so.major.minor" and
binpac was previously letting the post-release number into the name
like "libbinpac.so.0.54-7", which isn't compatible with that scheme.

Related to zeek/zeek#649
@jsiwek

This comment has been minimized.

Copy link
Member

@jsiwek jsiwek commented Nov 7, 2019

@douglas-carmichael thanks, a couple things going on:

  • The linking error should be fixed in zeek/master now
  • The --with-caf issue looks like it will require fixing on the CAF-side of things, too, so manually replacing Zeek's CAF with the topic/openbsd branch like you mentioned may be what you have to do for a bit
@douglas-carmichael

This comment has been minimized.

Copy link
Author

@douglas-carmichael douglas-carmichael commented Nov 7, 2019

What specific CAF fixes need to be made?

@jsiwek

This comment has been minimized.

Copy link
Member

@jsiwek jsiwek commented Nov 7, 2019

To fix -with-caf, I don't know what final fix CAF will go with. You can see my description of the issue at actor-framework/actor-framework#955 that offers one option to fix it could be a patch like this:

diff --git a/libcaf_core/CMakeLists.txt b/libcaf_core/CMakeLists.txt
index b051c857..ff2abe35 100644
--- a/libcaf_core/CMakeLists.txt
+++ b/libcaf_core/CMakeLists.txt
@@ -178,7 +178,7 @@ if (NOT CAF_BUILD_STATIC_ONLY)
   set_target_properties(libcaf_core_shared
     PROPERTIES
     SOVERSION ${CAF_VERSION}
-    VERSION ${CAF_VERSION}
+    VERSION ${CAF_VERSION_MAJOR}.${CAF_VERSION_MINOR}
     OUTPUT_NAME caf_core
   )
   install(TARGETS libcaf_core_shared
diff --git a/libcaf_io/CMakeLists.txt b/libcaf_io/CMakeLists.txt
index f82ccc4e..9e787e6d 100644
--- a/libcaf_io/CMakeLists.txt
+++ b/libcaf_io/CMakeLists.txt
@@ -64,7 +64,7 @@ if (NOT CAF_BUILD_STATIC_ONLY)
   set_target_properties(libcaf_io_shared
                         PROPERTIES
                         SOVERSION ${CAF_VERSION}
-                        VERSION ${CAF_VERSION}
+                        VERSION ${CAF_VERSION_MAJOR}.${CAF_VERSION_MINOR}
                         OUTPUT_NAME caf_io)
   install(TARGETS libcaf_io_shared
           RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
diff --git a/libcaf_opencl/CMakeLists.txt b/libcaf_opencl/CMakeLists.txt
index 33033bf6..f7c0ef60 100644
--- a/libcaf_opencl/CMakeLists.txt
+++ b/libcaf_opencl/CMakeLists.txt
@@ -28,7 +28,7 @@ if(NOT CAF_BUILD_STATIC_ONLY)
   set_target_properties(libcaf_opencl_shared
                         PROPERTIES
                         SOVERSION "${CAF_VERSION}"
-                        VERSION "${CAF_VERSION}"
+                        VERSION "${CAF_VERSION_MAJOR}.${CAF_VERSION_MINOR}"
                         OUTPUT_NAME caf_opencl)
   install(TARGETS libcaf_opencl_shared LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR})
 endif()
diff --git a/libcaf_openssl/CMakeLists.txt b/libcaf_openssl/CMakeLists.txt
index a0b3a3be..07640cea 100644
--- a/libcaf_openssl/CMakeLists.txt
+++ b/libcaf_openssl/CMakeLists.txt
@@ -32,7 +32,7 @@ if (NOT CAF_BUILD_STATIC_ONLY)
   set_target_properties(libcaf_openssl_shared
                         PROPERTIES
                         SOVERSION ${CAF_VERSION}
-                        VERSION ${CAF_VERSION}
+                        VERSION ${CAF_VERSION_MAJOR}.${CAF_VERSION_MINOR}
                         OUTPUT_NAME caf_openssl)
   if (CYGWIN)
     install(TARGETS libcaf_openssl_shared RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})
@douglas-carmichael

This comment has been minimized.

Copy link
Author

@douglas-carmichael douglas-carmichael commented Nov 7, 2019

I finally got zeek to compile, but got an error starting zeekctl:

dc-zeek# zeekctl
python3.7:/usr/local/lib/libbroker.so.2: undefined symbol '__inet_ntop'
python3.7:/usr/local/lib/libbroker.so.2: undefined symbol '__inet_pton'
Hint: Run the zeekctl "deploy" command to get started.

Welcome to ZeekControl 2.0.0-18

Type "help" for help.
[ZeekControl] >


Also, when I used "deploy" zeek complained about not finding eth0 in stderr.log:

fatal error: problem with interface eth0 (pcap_error: BIOCSETIF failed: Device not configured (pcap_activate))

Perhaps a check for a BSD platform and prompting the user for what interface to use?

@0xxon

This comment has been minimized.

Copy link
Member

@0xxon 0xxon commented Nov 7, 2019

I cannot speak to the inet_ntop error - perhaps @jsiwek has an idea.

As for the eth0 error - zeekctl requires the user to edit the configuration files to specify the listen device - this is true for all operating systems.

The default configuration file specifies eth0 - which may or may not be the correct choice on some linux systems. We currently don't provide any utilities to automatically edit these configuration files - that is up to the user.

@jsiwek

This comment has been minimized.

Copy link
Member

@jsiwek jsiwek commented Nov 9, 2019

The undefined symbol messages for inet_ntop/inet_pton should be addressed by #679 (also mention a few other caveats of general OpenBSD support status there).

@jsiwek jsiwek removed their assignment Nov 9, 2019
@Neverlord

This comment has been minimized.

Copy link
Member

@Neverlord Neverlord commented Nov 11, 2019

CAF master should work fine now on OpenBSD.

this also depends on patching CAF (actor-framework), we'd also need a 0.16.5 release of CAF which includes the other OpenBSD-support patches (@Neverlord could give more an opinion on plausibility of that).

I've backported the OpenBSD fixes to the 0.16 series. You can find that version in the branch topic/0.16.5.

If this is working for you then I'll tag 0.16.5.

@jsiwek

This comment has been minimized.

Copy link
Member

@jsiwek jsiwek commented Nov 11, 2019

caf/master and caf/topic/0.16.5 both checked out fine in my testing, thanks @Neverlord

@Neverlord

This comment has been minimized.

Copy link
Member

@Neverlord Neverlord commented Nov 11, 2019

@jsiwek Thank you for the quick feedback! As promised, CAF 0.16.5 is live, as well as CAF 0.17.3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Release 3.1.0
  
Unassigned / Todo
5 participants
You can’t perform that action at this time.