PLAN: replace MK with ooni/probe-engine

[bassosimone]

This plan has been drafted during the 17 February 2020 week. We want to keep it open until we have finished the migration and fully clarified that Measurement Kit is deprecated.

TL;DR

We will replace measurement-kit/measurement-kit (C++14) with ooni/probe-engine (Go). We will use gomobile bind to generate bindings for Go code. We will generate a C library containing an API that is ABI compatible with the existing FFI API exposed by Measurement Kit. The miniooni binary produced by ooni/probe-engine happens to have mostly the same CLI interface of measurement_kit.

Introduction

On mobile OONI uses Measurement Kit (MK). Specifically we use:

• https://github.com/measurement-kit/android-libs on Android
• https://github.com/measurement-kit/mkall-ios on iOS

They respectively expose a Java and ObjectiveC API. They do so by using MK's FFI API, which is implemented in MK's main repository: https://github.com/measurement-kit/measurement-kit. So, we need to discuss how we're going to switch over to ooni/probe-engine for mobile.

On desktop we use a different strategy. Since https://github.com/ooni/probe-cli is also written in Go, we link https://github.com/measurement-kit/measurement-kit into https://github.com/ooni/probe-engine using MK's FFI API. Thus, on desktop we will do the Ship-of-Theseus dance inside probe-engine. So, we don't need to discuss this issue further, since there is a smooth and clear upgrade path already.

There are also potentially users of the measurement_kit binary. While we don't know whether there are users of this binary, we need to discuss this use case, at least in theory.

Mobile clients

Mobile clients will use an API that is functionally equivalent to mkall-ios and android-libs, except that names will have a different prefix (e.g. MKFoo => OONIFoo). This will allow Go and C/C++ code to be linked with the same app. See ooni/probe-engine#347 for a description of how we've have reimplemented MKAsyncTask, i.e. the mobile API for running experiments.

We will keep MK code and Go code side by side. At the end of the process, we will pull the plug to MK code and have the apps only using Go code. The following diagram shows how the code structure will look like for OONI's Android app, for which we have started this integration process:

+--------------------------------------------------+
|          github.com/ooni/probe-android           |
+--------------------------------------------------+
| ooni/probe-engine | measurement-kit/android-libs | 
+-------------------+------------------------------+
                    |        measurement-kit       |
                    +------------------------------+

We have experimentally started integrating the AAR generated from Go code in ooni/probe-android#294. This is what changes in terms of app size:

Because the increase in the app size is bearable, we have the opportunity to choose whether we want to wait until all code is migrated in Go before releasing a new version, or proceeding piecemeal.

iOS is not discussed here because we want to focus on Android first, and not open too many PRs. The underlying technology (go mobile) is really expected to work in iOS just like Android. Hence, we roughly expect that for iOS we are going to follow a very similar procedure.

FFI clients

Clients using the FFI API will still use Measurement Kit. We will write a small Go wrapper around the mobile API that exports the FFI API. This will be done in this repository. (See the appendix for a prototypical implementation of this concept.)

MK will effectively become a C compatible API around ooni/probe-engine. The cost of doing this is small because the mobile API is already 99% of the FFI API. This will become the v0.11.x branch.

The v0.11.x branch will remain experimental until all the MK experiments will be rewritten in Go. At this point it will become the new master and master will become a legacy v0.10.x branch.

We will continue to maintain the v0.10.x branch until a specific EOL. This EOL will probably be at the beginning of Q3 2020, when OONI will have been fully migrated to probe-engine.

measurement_kit binary

These users can use the miniooni command produced by https://github.com/ooni/probe-engine. This command was designed to have the same CLI of measurement_kit for QA reasons.

Appendix

This is a prototypical implementation of the C FFI API:

package main

import (
  "C"
  "sync"

  "github.com/ooni/probe-engine/oonimkall"
)

var (
  idx int64 = 1
  m         = make(map[int64]*oonimkall.Task)
  mu        sync.Mutex
)

//export ooni_task_start
func ooni_task_start(settings string) int64 {
  tp, err := oonimkall.StartTask(settings)
  if err != nil {
    return 0
  }
  mu.Lock()
  handle := idx
  idx++
  m[handle] = tp
  mu.Unlock()
  return handle
}

//export ooni_task_interrupt
func ooni_task_interrupt(handle int64) {
  mu.Lock()
  if tp := m[handle]; tp != nil {
    tp.Interrupt()
  }
  mu.Unlock()
}

//export ooni_task_is_done
func ooni_task_is_done(handle int64) bool {
  isdone := true
  mu.Lock()
  if tp := m[handle]; tp != nil {
    isdone = tp.IsDone()
  }
  mu.Unlock()
  return isdone
}

//export ooni_task_wait_for_next_event
func ooni_task_wait_for_next_event(handle int64) (event string) {
  mu.Lock()
  tp := m[handle]
  mu.Unlock()
  if tp != nil {
    event = tp.WaitForNextEvent()
  }
  return
}

func main() {}

[bassosimone]

With ooni/probe-engine#492 this plan now mostly implemented. The only part that remains is this TODO list here: #1921.

[bassosimone]

Every detail has been nailed out quite nicely. The plan is good, execution in progress.