Skip to content
Sonja Heinze edited this page Dec 4, 2023 · 1 revision

The Fuzzy CI is a CI on Merlin that catches regressions by checking the responses of a fixed set of ocamlmerlin queries. That set of queries is determined by a deterministic but randomly chosen set of file location samples on the Irmin code base. The CI passes automatically if the responses on all samples remain the same before and after the PR. If there's a diff in the responses, the CI fails and makes the diff available as a GH action artifact.

If the diff looks good to the maintainers, they approve it by setting the fuzzy-diff-looks-good label on the PR, making the CI pass. With that, the CI comments on the PR with two pieces of information: the state of approval (a sha256 hash of the approved diff) and the diff download URL. If the PR is approved via that label and the state of the diff remains unchanged in future PR updates, the CI passes automatically.

The CI runs ocamlmerlin on over 25000 samples, so reading the entire diff can be tedious. To help, the CI additionally provides the diff of a simplified/"distilled" version of the responses, which breaks each response down into the following categories:

  1. The kind of response, i.e. one of exception, error, failure, a return containing a (error) message, a return containing an empty list, or a return containing anything else.
  2. The query number. If it becomes 0, then the ocamlmerlin-server has crashed.

About the implementation: The response data is generated in the CI, as opposed to living in the repo, for two reasons: 1. The data is over 20 MB big. 2. The generation of the data takes over 10 minutes, and we don't want to ask our contributors to re-generate it when opening certain PRs.