ooniprobe: new packages #11116

ja-pa · 2020-01-25T13:57:50Z

Maintainer: me
Compile tested: Turris Omnia (TOS5), OpenWrt master
Run tested: Turris Omnia (TOS5), OpenWrt master

Description:
This PR adds ~~two packages probe-cli~~ ooniprobe package related to network measurements for OONI (Open Observatory of Network Interference).
It helps researchers collect information about malicious behavior in the public networks (us of DPI, middleboxes, censorship,etc.)

Project website https://ooni.org/post/ https://ooni.org

ja-pa · 2020-01-25T17:20:48Z

The compilation is failing because https://github.com/Psiphon-Labs/bolt doesn't detect mipsel arch. (related issue boltdb/bolt#656) Will try to fix that in probe-engine pkg

BKPepe · 2020-01-25T22:39:13Z

Wouldn't help if you are going to create an issue in both repositories of the packages, which you want to add to ask if they are going to use bbolt instead of a bolt?

In probe-cli/go.sum and same for probe-engine/go.sum , they are using:
github.com/Psiphon-Labs/bolt v0.0.0-20190731171712-94750aa2185e h1:2zxppKeojJOuiO2aRjvVTD13hTPb+0RCia+TbGZX4Dg= and
github.com/boltdb/bolt v1.3.1 h1:JQmyP4ZBrce+ZQu0dY660FMfatumYDLun9hBCUVIkF4=

In repository Psiphon-Labs/bolt, I see the last commit on 31st July 2019. However, in bolt repository, I was able to found this:

If you are interested in using a more featureful version of Bolt, I suggest that you look at the CoreOS fork called bbolt.

So, you can create an issue in Psiphon-Labs/bolt to add support for mips. As I would like to see handled this in upstream instead of patching it here.

ja-pa · 2020-01-28T16:39:38Z

Thx, I created PR to fix bolt Psiphon-Labs/bolt#1 But still, it will take time when developers will integrate that to upstream version of the package because probe-cli depends on quite an old version of bolt.
I also squash probe-engine into probe-cli, because there is no need to compile it separately and it also fixes some issue with linking library.

ja-pa · 2020-02-03T13:41:38Z

I squashed commits into a single one. The package contains src for probe-engine and bolt package, which needs to be patched for OpenWrt. After a new probe-cli release, I believe it will be possible to remove these patches.

BKPepe · 2020-02-03T13:46:27Z

@jefferyto may I ask you to review this Go package?

net/probe-cli/Makefile

jefferyto · 2020-02-03T17:30:23Z

net/probe-cli/patches-probe-engine/001-fix-cgo.patch

+	// #cgo LDFLAGS: -levent_core
+	// #cgo LDFLAGS: -levent_extra
+	// #cgo LDFLAGS: -levent_pthreads
+	// #cgo LDFLAGS: -lz


Can this patch be upstreamed?

I created an issue , but it's not their priority right now.

Would it work if you added these flags to TARGET_LDFLAGS instead of patching this file?

I tried that, but it failed in the liking phase. arch Linux has similar patch

https://github.com/ooni/probe-engine/blob/master/measurementkit/task_cgo.go#L29

I've added CGO_LDFLAGS in #11255. That should allow you to set TARGET_LDFLAGS and have it affect cgo compilation (instead of having to patch the flags into the file).

I suggest patching out the #cgo linux,amd64 LDFLAGS: lines though, since that will affect x86-64 builds.

@bassosimone Ok, so I replaced default sentry url with my own url and get the same error as you.

I got this error when testing in malta-be (but not armvirt-64, although I didn't let it finish the whole test). I don't have a stack trace to compare though.

Perhaps you can try running upstream's pre-built (static) binary to see if you get the same error? If it doesn't happen with their binary then it's probably something in our build.

I just tested version rc9 to rc3 on OpenWrt. The same error is in 9,8,7,6,5,4 but version 3 works as expected. Upstream version for amd64 works without panic.

Just want to confirm: Did you also change the version of probe-engine used when testing the different versions?

Yes (more or less) I removed the part with probe-engine, since after your changes to golang its no longer necessary. I will push the updated package.

net/probe-cli/patches-probe-engine/003-fix-sum.patch

jefferyto · 2020-02-06T22:01:27Z

Also, perhaps this package can be named ooniprobe or ooniprobe-cli (or something with "ooni" in it)? "probe-cli" seems a bit too generic to me.

bassosimone · 2020-02-10T16:20:10Z

Wouldn't help if you are going to create an issue in both repositories of the packages, which you want to add to ask if they are going to use bbolt instead of a bolt?

I asked the same question some time ago. I was told that Psiphon prefers to use their own fork of bolt because they have developed a bunch of patches for robustness that they'd rather keep.

(OONI does not use bolt directly.)

bassosimone · 2020-02-10T16:20:49Z

Also, perhaps this package can be named ooniprobe or ooniprobe-cli (or something with "ooni" in it)? "probe-cli" seems a bit too generic to me.

Yes, please! (FWIW, I like ooniprobe the most, given that it seems to me unlikely that OpenWrt will package the Desktop app, and also the binary is called ooniprobe.)

hellais · 2020-02-10T16:32:50Z

Let me also join the package naming game party 🎉

Maybe I suggest calling the package: ooniprobe?

That would be consistent with the naming OONI Probe packages for Command Line Interface users have always had.
Since the CLI binary tool is still invoked via ooniprobe I think that should also be the name of the package.

The reason why the source code repo is called ooni/probe-cli, is that we now have version of OONI Probe for Desktop and Mobile as well, hence the distinctive naming.

Thanks so much for helping out with packaging it!

hellais · 2020-02-10T16:36:07Z

Another very useful thing to add to the package would be some way for people to run OONI Probe automatically on a regular basis.

I am not very familiar with what is generally the best practice for doing this in OpenWRT land, but something like a simple cronjob should be enough.

This can either be enabled by default or something the users opt-in to doing.

Check the optional section of the probe-cli readme for a linux and macOS example for this: https://github.com/ooni/probe-cli#user-setup.

jefferyto · 2020-02-14T01:20:54Z

You still need to patch out these lines in probe-engine (or somehow set CGO_LDFLAGS_DISALLOW to exclude those flags), otherwise they will affect compilation for x86-64 (linux,amd64).

bassosimone · 2020-02-17T07:47:53Z

@ja-pa I have addressed the issue and now you probably don't need to patch sources anymore. Could you perhaps see if you are satisfied with the new code? ooni/probe-engine#333 🙏

ja-pa · 2020-05-07T12:09:13Z

@jefferyto I think that measurement-kit is not necessary for runtime (I just checked that on my Omnia).

ja-pa · 2020-05-12T12:41:55Z

Updated to version 3.0.0

net/ooniprobe/Makefile

neheb · 2020-05-13T22:16:53Z

measurement_kit needs a dependency on libevent2-openssl.

ja-pa · 2020-05-14T14:36:02Z

measurement_kit needs a dependency on libevent2-openssl.
Yes, but here compilation fails without it

Package ooniprobe is missing dependencies for the following libraries:
libevent_openssl-2.1.so.7

jefferyto · 2020-05-19T09:53:16Z

net/ooniprobe/Makefile

+  DEPENDS:=\
+	$(GO_ARCH_DEPENDS) +libevent2-openssl \
+	+libevent2-core +libevent2-extra +libevent2-pthreads \
+	+libmaxminddb +libnghttp2 +libopenssl \


Are you sure libnghttp2 is necessary?

Without it I get

Package ooniprobe is missing dependencies for the following libraries: libnghttp2.so.14

I have compiled this package without +libnghttp2 in DEPENDS (and without the -lnghttp2 in TARGET_LDFLAGS) in armvirt-32 and x86-generic, and I haven't encountered this error... Is there a specific target arch I should try compiling for to see this error?

I use mvebu which should be arm32 . I think it could be somehow related to ooni/probe-engine@ae59c05#diff-38f5d43a0af2601dc205a09e9e321845

When we compile for Linux without the "ooni" flag (which does what we want in our strange Linux setup) we just -lcurl and assume the dynamic linker will figure out.

When I did that, I assumed a distro would have preferred dynamic linking. Is there a reason not to use dynamic linking of C/C++ dependencies here?

Dynamic linking to curl here is fine; TARGET_LDFLAGS has -lcurl.

I was asking about libnghttp2 - it doesn't appear to be necessary anymore in the latest release.

So, MK never depended directly from nghttp2. AFAIK, however, curl uses nghttp2 to implement http2. Hence, whether it's needed or not I think depends on libcurl.so?

neheb · 2020-05-31T23:03:58Z

What's the status on this?

jefferyto · 2020-06-02T09:37:29Z

I'm don't believe libnghttp2 is a run-time dependency, or perhaps it should be selected if LIBCURL_NGHTTP2 is selected (similar to libcurl).

jefferyto · 2020-07-10T08:47:24Z

net/ooniprobe/Makefile

+  DEPENDS:=\
+	$(GO_ARCH_DEPENDS) +libevent2-openssl \
+	+libevent2-core +libevent2-extra +libevent2-pthreads +libnghttp2 \
+	+libmaxminddb +libopenssl +LIBCURL_NGHTTP2:libnghttp2 \


libnghttp2 appears twice in the above 2 lines.

jefferyto · 2020-07-10T08:48:05Z

net/ooniprobe/Makefile

+
+TARGET_LDFLAGS:=\
+	-lmeasurement_kit -lmaxminddb -lcurl \
+	-lnghttp2 -levent_openssl -lssl \


I believe -lnghttp2 also depends on LIBCURL_NGHTTP2 being set.

I removed that and build/run test worked file

jefferyto · 2020-07-22T19:25:15Z

I think the CI error is because the measurement-kit package needs to be updated to 0.10.12.

ja-pa · 2020-07-22T23:29:04Z

@jefferyto Thx, I rebased ooni branch to master

bassosimone · 2020-07-23T08:53:29Z

@ja-pa @jefferyto as a heads-up, this is one of the latest ooni/probe-cli that uses Measurement Kit as a dependency. Also, as another heads-up: we've discovered an issue with 3.0.4 and we're cutting 3.0.5 in ~1 hour to fix the issue.

ja-pa · 2020-08-03T17:11:14Z

I just updated ooniprobe to version 3.0.6 and removed dependency on libnghttp2

Runtested with im tests

jefferyto · 2020-08-04T19:24:01Z

I'd like to have this added:

define Package/ooniprobe/config
  select MEASUREMENT_KIT_BUILD_DEPENDS
endef

but otherwise this looks good to me.

ja-pa · 2020-08-04T22:30:43Z

I'd like to have this added:
define Package/ooniprobe/config
  select MEASUREMENT_KIT_BUILD_DEPENDS
endef
but otherwise this looks good to me.

Added, thanks !

Signed-off-by: Jan Pavlinec <jan.pavlinec@nic.cz>

jefferyto

Thanks for your work on this!

ja-pa changed the title ~~probe-cli & probe-engine: new packages~~ WIP: probe-cli & probe-engine: new packages Jan 25, 2020

ja-pa force-pushed the ooni branch 3 times, most recently from 82b7b72 to 6aaa83a Compare January 27, 2020 09:16

ja-pa force-pushed the ooni branch from 9c953db to 31b531c Compare February 3, 2020 13:36

ja-pa changed the title ~~WIP: probe-cli & probe-engine: new packages~~ probe-cli: new packages Feb 3, 2020

jefferyto reviewed Feb 3, 2020

View reviewed changes

ja-pa force-pushed the ooni branch from 4779f31 to 81a3873 Compare February 4, 2020 11:34

jefferyto reviewed Feb 5, 2020

View reviewed changes

net/probe-cli/patches-probe-engine/003-fix-sum.patch Outdated Show resolved Hide resolved

ja-pa force-pushed the ooni branch from 241e7cf to 9670953 Compare February 5, 2020 10:58

ja-pa changed the title ~~probe-cli: new packages~~ WIP probe-cli: new packages Feb 6, 2020

ja-pa force-pushed the ooni branch from bbce1cc to 5d16f9c Compare February 13, 2020 21:42

ja-pa changed the title ~~WIP probe-cli: new packages~~ WIP ooniprobe ~~probe-cli~~: new packages Feb 17, 2020

ja-pa changed the title ~~WIP ooniprobe ~~probe-cli~~: new packages~~ WIP ooniprobe: new packages Feb 17, 2020

ja-pa mentioned this pull request Feb 17, 2020

task_cgo.go: allow compiling on ordinary Linux systems ooni/probe-engine#333

Merged

ja-pa force-pushed the ooni branch 2 times, most recently from c922571 to 6750b86 Compare April 29, 2020 12:28

ja-pa changed the title ~~WIP ooniprobe: new packages~~ ooniprobe: new packages Apr 29, 2020

ja-pa force-pushed the ooni branch from 9e039aa to 42c856a Compare May 12, 2020 12:40

neheb reviewed May 13, 2020

View reviewed changes

net/ooniprobe/Makefile Show resolved Hide resolved

neheb reviewed May 13, 2020

View reviewed changes

net/ooniprobe/Makefile Show resolved Hide resolved

jefferyto reviewed May 19, 2020

View reviewed changes

ja-pa mentioned this pull request May 19, 2020

measurement-kit: Add static build dependencies #12217

Merged

ja-pa force-pushed the ooni branch 3 times, most recently from cfdfa6d to 6287476 Compare June 16, 2020 12:27

ja-pa force-pushed the ooni branch from ae135d6 to ff1eeb6 Compare July 9, 2020 13:38

jefferyto reviewed Jul 10, 2020

View reviewed changes

ja-pa force-pushed the ooni branch from c1318fe to 410f6b0 Compare July 22, 2020 11:52

ja-pa force-pushed the ooni branch from 410f6b0 to 11c64a3 Compare July 22, 2020 23:28

ja-pa force-pushed the ooni branch 2 times, most recently from 91f754c to 3b740b0 Compare August 3, 2020 16:54

ja-pa force-pushed the ooni branch from 3b740b0 to a588b44 Compare August 4, 2020 22:26

ooniprobe: add new package

a588b44

Signed-off-by: Jan Pavlinec <jan.pavlinec@nic.cz>

jefferyto approved these changes Aug 5, 2020

View reviewed changes

neheb merged commit c4254a3 into openwrt:master Aug 5, 2020

ooniprobe: new packages #11116

ooniprobe: new packages #11116

Conversation

ja-pa commented Jan 25, 2020 • edited

ja-pa commented Jan 25, 2020 • edited

BKPepe commented Jan 25, 2020

ja-pa commented Jan 28, 2020

ja-pa commented Feb 3, 2020

BKPepe commented Feb 3, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ja-pa Feb 13, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ja-pa Feb 13, 2020 • edited

Choose a reason for hiding this comment

jefferyto commented Feb 6, 2020

bassosimone commented Feb 10, 2020

bassosimone commented Feb 10, 2020 • edited

hellais commented Feb 10, 2020

hellais commented Feb 10, 2020

jefferyto commented Feb 14, 2020

bassosimone commented Feb 17, 2020

ja-pa commented May 7, 2020

ja-pa commented May 12, 2020

neheb commented May 13, 2020

ja-pa commented May 14, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jefferyto May 19, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bassosimone May 21, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neheb commented May 31, 2020

jefferyto commented Jun 2, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jefferyto commented Jul 22, 2020

ja-pa commented Jul 22, 2020

bassosimone commented Jul 23, 2020

ja-pa commented Aug 3, 2020 • edited

jefferyto commented Aug 4, 2020

ja-pa commented Aug 4, 2020

jefferyto left a comment

Choose a reason for hiding this comment

ja-pa commented Jan 25, 2020 •

edited

ja-pa commented Jan 25, 2020 •

edited

ja-pa Feb 13, 2020 •

edited

ja-pa Feb 13, 2020 •

edited

bassosimone commented Feb 10, 2020 •

edited

ja-pa commented May 14, 2020 •

edited

jefferyto May 19, 2020 •

edited

bassosimone May 21, 2020 •

edited

ja-pa commented Aug 3, 2020 •

edited