Since I wrote about the new ATProto-powered Tangled Git forge a few months ago, it's come along by leaps and bounds!
+First, and most excitingly, they've added continuous integration via Spindles which are built in a nice ATProto style:
+++When you push code or open a pull request, the knot hosting your repository +emits a pipeline event (sh.tangled.pipeline). Running as a dedicated service, +spindle subscribes to these events via websocket connections to your knot.
+
The pipelines are Nix-only right now, so I braved using it[1] for a new GPS Exchange Format library in OCaml that I wrote. The pipelines should look familiar, and the description format very straightforward.
+Secondly, the service has added support for JJ stacked pull requests, which are the closest I've seen to the Jane Street Iron diff workflow which I've been wanting to try in open source for ages. You can see the interdiff review process on a recent PR by Winter who add support for engine-agnostic Spindle workflows, which should pave the path for a Docker or BuildKit engine alongside the existing Nixery-based one.
+And thirdly, the general quality-of-life of the web frontend has improved dramatically, with a nice timeline, repo list, and profile pages. I'm running two knots right now (one on Recoil, and one in the Cambridge Computer Lab), and both have been very painfree. I wrote one of the earliest Dockerfiles for it, but there's now a community-maintained Knot Docker setup which I've switched to. Doesn't take very long at all; give it a try!
+Because I've been using Tangled so much, I added support for Tangled metadata to Dune to make OCaml package maintainence easier. This will appear in Dune 3.21 in a few months, but in the meanwhile enjoy the first Tangled.sh package on opam. It's a simple GPX library I used in my recent trip to Botswana. All you need in your dune-project will be:
(lang dune 3.21)
+(name mlgpx)
+(generate_opam_files true)
+(source (tangled @anil.recoil.org/ocaml-gpx))
+The only major thing I'm missing from Tangled is support for private repositories now, but I'm very content using it for public content today. Beware as usual that it's still in alpha, so don't trust super-ultra-mega-important stuff to it unless you've git mirrored elsewhere.
+An OCaml MCP server
LLMs are proving themselves superbly capable of a variety of coding tasks, having been trained against the enormous amount of code, tutorials and manuals available online. However, with smaller languages like OCaml there simply isn't enough training material out there, particularly when it comes to new language features like effects or new packages that haven't had time to be widely used. With my colleagues Anil, Ryan and Sadiq we've been exploring ways to improve this situation. One way we can mitigate these challenges is to provide a Model Context Protocol (MCP) server that's capable of providing up-to-date info on the current state of the OCaml world.
The MCP specification was released by Anthropic at the end of last year. Since then it has become an astonishingly popular mechanism for extending the capabilities of LLMs, allowing them to become incredibly powerful agents capable of much more than simply chatting. There are now a huge variety of MCP servers, from one that provides professional cycling data to one that can do your email. The awesome mcp server list already lists hundreds, and these are just the awesome ones!
I've been working with Sadiq to make an MCP server for OCaml, with an initial focus on building it such that it can be hosted for everyone rather than something that is run locally. Our plan is to start with a service that can help with choosing OCaml libraries, by taking advantage of the work done by ocaml-docs-ci which is the tool used to generate the documentation for all packages in opam-repository and is served by ocaml.org. As well as producing HTML docs, we can also extract a number of other formats from the pipeline, including a newly created markdown backend. Using this, we can get markdown-formatted documentation for the every version of every package in the OCaml ecosystem.
Continue reading here
diff --git a/data/planet/robur.coop/a-web-ui-for-dnsvizor.md b/data/planet/robur.coop/a-web-ui-for-dnsvizor.md new file mode 100644 index 0000000000..33218ddb77 --- /dev/null +++ b/data/planet/robur.coop/a-web-ui-for-dnsvizor.md @@ -0,0 +1,15 @@ +--- +title: A Web UI for DNSvizor +description: +url: https://blog.robur.coop/articles/dnsvizor05.html +date: 2025-06-18T00:00:00-00:00 +preview_image: +authors: +- Robur Cooperative +source: +ignore: +--- + + + The NGI-funded DNSvizor provides core network services on your network; DNS resolution and DHCP. We added a web UI and metrics. + diff --git a/data/planet/robur.coop/authentication-and-extended-web-ui-for-dnsvizor.md b/data/planet/robur.coop/authentication-and-extended-web-ui-for-dnsvizor.md new file mode 100644 index 0000000000..4769b5bbf5 --- /dev/null +++ b/data/planet/robur.coop/authentication-and-extended-web-ui-for-dnsvizor.md @@ -0,0 +1,15 @@ +--- +title: Authentication and extended web UI for DNSvizor +description: +url: https://blog.robur.coop/articles/dnsvizor08.html +date: 2025-07-22T00:00:00-00:00 +preview_image: +authors: +- Robur Cooperative +source: +ignore: +--- + + + DNSvizor now has authentication mechanisms for users who wish to update their block lists or modify configuration data etc. + diff --git a/data/planet/robur.coop/dnsvizor---run-your-own-dhcp-and-dns-mirageos-unikernel---gets-dnssec-support.md b/data/planet/robur.coop/dnsvizor---run-your-own-dhcp-and-dns-mirageos-unikernel---gets-dnssec-support.md new file mode 100644 index 0000000000..8e14158613 --- /dev/null +++ b/data/planet/robur.coop/dnsvizor---run-your-own-dhcp-and-dns-mirageos-unikernel---gets-dnssec-support.md @@ -0,0 +1,15 @@ +--- +title: DNSvizor - run your own DHCP and DNS MirageOS unikernel - gets DNSsec support +description: +url: https://blog.robur.coop/articles/dnsvizor03.html +date: 2025-05-24T00:00:00-00:00 +preview_image: +authors: +- Robur Cooperative +source: +ignore: +--- + + + The NGI-funded DNSvizor provides core network services on your network; DNS resolution and DHCP. + diff --git a/data/planet/robur.coop/dnsvizor-gets-blocklists.md b/data/planet/robur.coop/dnsvizor-gets-blocklists.md new file mode 100644 index 0000000000..fdbd7b3df7 --- /dev/null +++ b/data/planet/robur.coop/dnsvizor-gets-blocklists.md @@ -0,0 +1,13 @@ +--- +title: DNSvizor Gets Blocklists +description: +url: https://blog.robur.coop/articles/dnsvizor07.html +date: 2025-06-25T00:00:00-00:00 +preview_image: +authors: +- Robur Cooperative +source: +ignore: +--- + +DNSvizor is extended to support blocking (malicious) domains. diff --git a/data/planet/robur.coop/dnsvizor-manual.md b/data/planet/robur.coop/dnsvizor-manual.md new file mode 100644 index 0000000000..3d2188b2d4 --- /dev/null +++ b/data/planet/robur.coop/dnsvizor-manual.md @@ -0,0 +1,13 @@ +--- +title: DNSvizor manual +description: +url: https://blog.robur.coop/articles/dnsvizor10.html +date: 2025-07-30T00:00:00-00:00 +preview_image: +authors: +- Robur Cooperative +source: +ignore: +--- + +Documenting DNSvizor: setup, how to build, configuration, updates diff --git a/data/planet/robur.coop/dnsvizor-performance-engineering.md b/data/planet/robur.coop/dnsvizor-performance-engineering.md new file mode 100644 index 0000000000..562fc5acd7 --- /dev/null +++ b/data/planet/robur.coop/dnsvizor-performance-engineering.md @@ -0,0 +1,13 @@ +--- +title: DNSvizor performance engineering +description: +url: https://blog.robur.coop/articles/dnsvizor09.html +date: 2025-07-24T00:00:00-00:00 +preview_image: +authors: +- Robur Cooperative +source: +ignore: +--- + +Evaluating and engineering the performance of DNSvizor diff --git a/data/planet/robur.coop/dnsvizor-with-dns-over-tls-and-dns-over-https-support.md b/data/planet/robur.coop/dnsvizor-with-dns-over-tls-and-dns-over-https-support.md new file mode 100644 index 0000000000..433dfce11d --- /dev/null +++ b/data/planet/robur.coop/dnsvizor-with-dns-over-tls-and-dns-over-https-support.md @@ -0,0 +1,15 @@ +--- +title: DNSvizor with DNS over TLS and DNS over HTTPS support +description: +url: https://blog.robur.coop/articles/dnsvizor04.html +date: 2025-05-29T00:00:00-00:00 +preview_image: +authors: +- Robur Cooperative +source: +ignore: +--- + + + The NGI-funded DNSvizor provides core network services on your network; DNS resolution and DHCP. We added support for DNS over TLS and DNS over HTTPS + diff --git a/data/planet/robur.coop/privacy-for-dnsvizor.md b/data/planet/robur.coop/privacy-for-dnsvizor.md new file mode 100644 index 0000000000..70e3cebea1 --- /dev/null +++ b/data/planet/robur.coop/privacy-for-dnsvizor.md @@ -0,0 +1,15 @@ +--- +title: Privacy for DNSvizor +description: +url: https://blog.robur.coop/articles/dnsvizor06.html +date: 2025-07-03T00:00:00-00:00 +preview_image: +authors: +- Robur Cooperative +source: +ignore: +--- + + + The NGI-funded DNSvizor provides core network services on your network; DNS resolution and DHCP. We improved the privacy. + diff --git a/data/planet/robur.coop/streaming-binary-data-in-mollymawk.md b/data/planet/robur.coop/streaming-binary-data-in-mollymawk.md new file mode 100644 index 0000000000..6621f2a484 --- /dev/null +++ b/data/planet/robur.coop/streaming-binary-data-in-mollymawk.md @@ -0,0 +1,15 @@ +--- +title: Streaming Binary Data in Mollymawk +description: +url: https://blog.robur.coop/articles/streaming-binaries-mollymawk.html +date: 2025-08-18T00:00:00-00:00 +preview_image: +authors: +- Robur Cooperative +source: +ignore: +--- + + + Mollymawk receives a brand new update that let's it stream binary data avoiding memory issues. + diff --git a/data/planet/robur.coop/streaming-console-output-in-mollymawk.md b/data/planet/robur.coop/streaming-console-output-in-mollymawk.md new file mode 100644 index 0000000000..dcb20e41fc --- /dev/null +++ b/data/planet/robur.coop/streaming-console-output-in-mollymawk.md @@ -0,0 +1,15 @@ +--- +title: Streaming console output in mollymawk +description: +url: https://blog.robur.coop/articles/mollymawk-console.html +date: 2025-05-21T00:00:00-00:00 +preview_image: +authors: +- Robur Cooperative +source: +ignore: +--- + + + HTTP server-sent events are very helpful for streaming live console data, we implemented that in mollymawk + diff --git a/data/planet/tarides/internship-report-refactoring-tools-coming-to-merlin.md b/data/planet/tarides/internship-report-refactoring-tools-coming-to-merlin.md new file mode 100644 index 0000000000..76f9e46dd6 --- /dev/null +++ b/data/planet/tarides/internship-report-refactoring-tools-coming-to-merlin.md @@ -0,0 +1,118 @@ +--- +title: 'Internship Report: Refactoring Tools Coming to Merlin' +description: "Tim\xE9o shares his project integrating new refactoring features into + Merlin, improving user experience with a new command." +url: https://tarides.com/blog/2025-08-20-internship-report-refactoring-tools-coming-to-merlin +date: 2025-08-20T00:00:00-00:00 +preview_image: https://tarides.com/blog/images/tools-merlin-1360w.webp +authors: +- Tarides +source: +ignore: +--- + +Refactoring features have contributed to the popularity of editors like IntelliJ, as well as certain programming languages whose editor support offers interactive mechanisms to manage code — Gleam being an excellent example. Even though OCaml has some features related to refactoring (such as renaming occurrences, substituting typed holes with expressions, and case analysis for pattern matching), the goal of my internship was to kickstart work on a robust set of features to enable the smooth integration of multiple complementary refactoring support commands.
+As part of my Tarides internship (on the editor side), I specified several useful commands, inspired by competitors and materialised in the form of RFCs, subject to discussion. There were multiple candidates, but we found that expression extraction to toplevel was the most suitable for a first experiment. Since it touched on several parts of the protocol and required tools that could be reused for other features, it was important to design the system with extensibility and modularity in mind.
+In this article, I will present the results of this experiment, including the new command and some interesting use cases.
+Examples
+Expression extraction to toplevel will select the most inclusive expression that fits in your selection and propose to extract it. In this case, extract means that the selected expression will be moved into its own freshly generated let binding top level.
Extracting Constants
+Here is a first example: Let's try to extract a constant. Let’s assume +that the float 3.14159 is selected in the following code snippet:
+let circle_area radius = 3.14159 *. (radius ** 2.)
+ (* ^^^^^^^ *)
+The extract action code will then be proposed, and if you apply it, the code will look like this:
let const_name1 = 3.14159
+let circle_area radius = const_name1 *. (radius ** 2.)
+Here is an illustrated example (based on an experimental branch of ocaml-eglot):
+
+
+
We can see that the expression has been effectively extracted and replaced by a reference to the fresh let binding. We can also observe that in the absence of a specified name, the generated binding will be named with a generic name that is not taken in the destination scope. You also have the ability to supply the name you want for extraction.
+For example, here is the same example where the user can enter a name:
+
+
+
But the refactoring capabilities go much further than constant extraction!
+Extracting an Expression
+In our previous example, we could speculate about the purity of the expression, since we were only extracting a literal value. However, OCaml is an impure language, so extracting an expression into a constant can lead to unintended behavior. For example, let's imagine the following snippet:
+let () =
+ let () =
+ print_endline "Hello World!";
+ print_endline "Done"
+ in ()
+In this example, extracting into a constant would cause problems! Indeed, we would be changing the semantics of our program by executing both print statements beforehand. Fortunately, the command analyses the expression as not being a constant and delays its execution using a thunk — a function of type unit -> ....
+
+
As we can see, our goal was to maximise the production of valid code, as much as possible, by carefully analysing how to perform the extraction. This is all the more challenging in OCaml, which allows for arbitrary (and potentially infinite) nesting of expressions.
+Extracting an Expression That Uses Variables
+The final point we’ll briefly cover is the most fun. Indeed, it’s possible that the expression we want to extract depends on values defined in the current scope. For example:
+let z = 45
+
+let a_complicated_function x y =
+ let a = 10 in
+ let b = 11 in
+ let c = 12 in
+ a + b + c + (c * x * y) + z
+In this example, the extraction of the expression a + b + c (c * x * y) + z will be placed between z and a_complicated_function. As a result, z will still be accessible; however, x, y, a, b, and c will be free variables in the extracted expression. Therefore, we generate a function that takes these free variables as arguments:
+
+
Identifying free variables was one of the motivations for starting with this command. We are fairly certain that this is a function that we will need to reuse in many contexts! Note that the command behaves correctly in the presence of objects and modules.
+A Real World Example
+Let’s try to extract something a little more complicated now. Let’s assume we have the following code and we want to refactor it, for example, by extracting the markup type pretty print logic outside our pp function.
type t = markup list
+and markup = Text of string | Bold of string
+
+let show doc =
+ let buf = Buffer.create 101 in
+ let bold_tag = "**" in
+ List.iter
+ (fun markup ->
+ Buffer.add_string buf
+ @@
+ match markup with
+ | Text txt -> txt
+ | Bold txt -> bold_tag ^ txt ^ bold_tag)
+ doc;
+ Buffer.contents buf
+We can observe that bounded variables in the extracted region are now passed as arguments, and the extracted function is properly replaced by a call to the new show_markup generated function.
+let show_markup buf bold_tag =
+ fun markup ->
+ (Buffer.add_string buf)
+ (match markup with
+ | Text txt -> txt
+ | Bold txt -> bold_tag ^ txt ^ bold_tag)
+let show doc =
+ let buf = Buffer.create 101 in
+ let bold_tag = "**" in
+ List.iter (show_markup buf bold_tag) doc;
+ Buffer.contents buf
+Here is an example of how it is used. Impressive, isn't it?
+
+
+
Editor Support
+To understand how this new Merlin command can be properly used in your favourite editor, we have to take a closer look at the functioning of the Language Server Protocol. The LSP supports two mechanisms to extend the existing protocol with new features. First, there is code action, which allows us to perform multiple LSP commands sequentially. This kind of request has the merit of working out of the box without requiring any plugin or specific command support on the editor side (which oils the wheels for maintenance). Secondly, there are custom requests, which are more powerful than code actions and enable custom interactivity. So, if you want to prompt the user, a custom request is the way to go. The price you have to pay for this power is to have client-side support implemented for each custom request in every editor plugin.
The current editor team approach is as follows: For each of Merlin's commands that don't map directly to a standard LSP request, we provide a code action associated with the Merlin command and potentially a dedicated custom request if the feature requires custom interactivity. Regarding the ‘extract’ feature, the associated code action does not allow us to choose the name of the generated let binding, but the custom request does.
+What’s Next?
+I hope this new command helps you get even more productive in OCaml! Don’t hesitate to experiment with it and report any bugs you encounter.
+The development of Merlin’s refactoring tools was part of a broader vision to improve OCaml editor support and perhaps claim an editor experience similar to JetBrains IDE in the future!
+The work done on the extract command gives us the opportunity to identify various problems pertaining to refactoring (substitution, code generation) and potentially to make the connection to refactoring commands that already exist in Merlin (like open refactoring and project-wide renaming). The next step is to add a small toolbox library in Merlin dedicated to refactoring in order to develop even more refactor actions. I hope this is just the first refactoring feature of a long series.
If you're curious and want to take a look at the feature, it's split into several PRs:
+-
+
- ocaml/merlin#1948 which implements the extraction logic on the Merlin side and exposes it in the protocol, +
- ocaml/ocaml-lsp#1545 which exposes the Custom Request enabling the use of the LSP-side functionality, +
- ocaml/ocaml-lsp#1546 which exposes an Action Code that allows the functionality to be invoked without additional formalities on the Editor side, +
- tarides/ocaml-eglot#65 which implements extraction behaviour in OCaml-Eglot, invocable either from a type enclosing or directly as a classic Emacs command. +
All of these PRs are currently under review, and should be merged soon!
+A big thanks to Xavier, Ulysse, and all the people that helped me during this internship. It was pretty interesting!
+ From 810fbc3147bbbc4f53df4fd517a5d83132fc482d Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Mon, 25 Aug 2025 08:17:10 +0200 Subject: [PATCH 07/15] [scrape.yml] New OCaml Planet blog posts and videos (#3271) Co-authored-by: cuihtlauac <1917592+cuihtlauac@users.noreply.github.com> --- ...gy-of-the-internet-ideas-at-aarhus-2025.md | 143 ++++++++++++++++++ 1 file changed, 143 insertions(+) create mode 100644 data/planet/anil/presenting-our-ecology-of-the-internet-ideas-at-aarhus-2025.md diff --git a/data/planet/anil/presenting-our-ecology-of-the-internet-ideas-at-aarhus-2025.md b/data/planet/anil/presenting-our-ecology-of-the-internet-ideas-at-aarhus-2025.md new file mode 100644 index 0000000000..b5b392fbbd --- /dev/null +++ b/data/planet/anil/presenting-our-ecology-of-the-internet-ideas-at-aarhus-2025.md @@ -0,0 +1,143 @@ +--- +title: Presenting our Ecology of the Internet ideas at Aarhus 2025 +description: +url: https://anil.recoil.org/notes/ecology-at-aarhus +date: 2025-08-22T00:00:00-00:00 +preview_image: +authors: +- Anil Madhavapeddy +source: +ignore: +--- + +That's a wrap for the next decade with Aarhus 2025, +where I presented our paper on "Steps towards an Ecology for the Internet". +I was a little unsure about how to approach the presentation, largely because +the ideas seem a little crazy if they'd been proposed even a year ago! Luckily +my co-authors strengthened my spine with encouragement and gin, +and the event was tremendous fun packed with useful insights.
+Our key observation is that the Internet is dangerously ossifying into monocultures +at multiple levels. Inspired by wild ecosystems, we're proposing mixing in more natural +selection +into edge deployments by using AI code models to mutate end-hosts and tailor +them to their environment. Generative AI is notoriously +unpredictable, +which turns out to be a useful property if you actually want more local +software diversity! For example, this lets us cook up "antibotty" networks that +fight back against global viruses via locally adapted vigilantes (antibodies).
+
Beyond just making thing more resilient, injecting more software diversity gives +us the hooks to make computers in our environment do what we actually want +them to. +To quote Mark Weiser, "the world is not a desktop", and computers +should be so ubiquitously blended into our day-to-day lives that they are +invisible to the user. We're +getting further away from that dream every day with the monotony of the FAANG +software monocultures, and a wild west of botnets +sweeping through billions of devices.
+Since this conference only happens once a decade, I put myself in the right mindframe by reading through my old ideas and seeing how they had aged. +I was struck by how much Databox (2015), Droplets (2011) and +Digital Yurts (2009) all stood up surprisingly well[1] in 2025. +This time around, we have a fresh edge with the rise of coding models and a relative glut of edge computation. The question is how to harness these new technologies for the health of the Internet and not yet more central lockin.
+
Discussions with the audience
+The talks were arranged in a panel with three other great speakers, who discussed crises through the lenses urban gardening, computing supply chains and two-loops models of change. There were some thought-provoking questions from the audience!
+Firstly, is framing these questions as a "crisis" just saturating us with a constant bombardment of problems we need to react to? Should we be building more (emotionally and systemically) sustainable platforms for engendering change? I certainly agreed with this, but I don't yet have a clear sense of what this means, beyond finding Our World In Data an inspiration of how fun data exploration can be.
+A couple of things I need to follow up on reading:
+-
+
- Computing as Ecocide was a paper at the LIMITS workshop, which I hadn't heard off before. This seems like a complement to PROPL coming up in October. +
- Robin Sloan is an author who combines old and new tech in fun ways. +
- Approaches to directly deal with "eco-anxiety" while teaching sustainability. +
Inversely proportional voting
+The stage discussion then veered into the role of lifecycles in these cultural +systems. An audience member asked about the role of biological mutualism and +cooperation in any future digital framework, and I pointed out +some +examples from our paper +about how cooperative ensembles in nature can be very stable, but also do not +have to last forever. When applying this "nihil aeternum est" principle to +human systems, how sacrosanct are concepts like "democracy" as we move forward? +If our views on these institutions remain unchanging, then they will also +become brittle and collapse as the context in which they operate changes into a hot +crowded world mired in polycrises.
+This let me bring out an idea I've been ruminating on for a while. While the principles of equal suffrage are vital, one dimension where we could relax it is for intergenerational representation. One issue with taking political decisions for the long-term is that wealth pools with the old, who have far less to gain from long-term thinking than the young[2] voters. So why are the young so underrepresented? Imagine a completely made up voting system that looked like this:
+++A voting system assigns each individual a number of votes inversely +proportional to their age. An 18-year old will have the maximum number of +votes, and they gradually degrade until anyone age 70 or higher gets just one +vote. The number of votes decay rapidly with age, so that every year the new +18-year old cohort will control (say) 40% of the total vote. This sort of +system ensures that the older generations (where wealth pools) must educate +the newly minted voters every year, or risk losing control of their agendas. +-- Anil's entirely made up voting system
+
I've found some variations of the theme, like Demeny voting that gives parents a proxy +vote for their children or even cases for disenfranchisement of the elderly. +But neither of these quite capture what I have in mind, which is to build +intergenerational education firmly into how our society operates. If 40% of +the entire voting block appears newly every year, then education on civic +matters need to happen like clockwork and be incorporated into our curriculums.
+But of course, there are huge barriers to trying out these experiments in civic +society. But there are no such barriers to running these experiments in +microcosms of common pool +resources +or even on digital systems. So after these fascinating conversations +at the conference, I'm going to think about the Cambridge Green Blue and apply it there.
+
Self hosting is really far behind
+I was also struck by how far behind self-hosting is, even among an audience +that should be heavily in favour of it. I think my talk was one of only a few +that mentioned BlueSky and the Fediverse, and alternative communication +mechanisms. I also demoed Claude in a corner to show how it could help manage +infrastructure that would +ordinarily take a sysadmin, but could now be reasonably handled by a non-expert +(with care!).
+One of the attendees commented to me afterwards that they remembered the +Databox talk from a decade ago, and wondered why it +hadn't taken off. Maybe now is the time for Ryan Gibb's work on digital +islands to hit the mainstream! +They've certainly never been needed more than now; I am deeply glad to +see my colleagues like Jon Sterling also working on solutions +in this space.
+Presenting in Slipshow
+In the spirit of self-hosting, I also used the great new
+Slipshow tool (that uses
+js_of_ocaml) to write the
+presentation. Slipshow lets the presentation be written in
+Markdown,
+and I used Claude Code to handle all the styling for me. The whole presentation
+took about an hour to put together, and can be viewed
+standalone as a single web page as it
+inlines all the assets.
Using HTML/JS/CSS for talks is really convenient, so I'm sold on using Slipshow
+for my upcoming presentations this year! It's also excellent to be using
+js_of_ocaml. I think the only thing on my "wishlist" is to be able to run a
+headless browser and output PDF snapshots of each of the slips. I'm also not
+yet sure how my 100MB videos will encode, but I'll figure that out ahead of my
+next talk (in the Royal Society at the start of September to the Austrian
+government). The author of the software, Paul-Elliot also
+kindly reached out to get feedback, and I was really pleased to see his work
+was supported by the NGI Commons Fund!
The city of Aarhus
+Aarhus is also a spiritual twin city to Cambridge. It was a gorgeously sunny +week, with bicycles available everywhere and a lovely Latin quarter to hang out +in.
+
The venue itself at Aarhus University was really nice to explore and see what +the students are up to. Lots of music and creative arts in the same area. When +I was here last year to present at +Matchpoints +we were at the Modern Art museum, and the amount of new building work in the +city was remarkable to see. There was also a lovely forest park I went for an +early morning jog in to get some nature!
+
-
+
-
+
Not coincidentally, most of these ideas were cooked up with Jon Crowcroft somewhere in the picture at a nearby pub.
+↩︎︎ -
+
We used the ideas of discount factors to adjust for impermanence in our forest carbon work in Nature Climate Change a few years ago too.
+↩︎︎
- Why Lean 4 replaced OCaml as my Primary Language
- Tacit 1.0.0 - Combinators!
- Slipshow!
- Claudius is now available in opam! 🎉
- New release of UnionFind
- Bogue, the OCaml GUI
- Dune 3.20
- cudajit: Bindings to the ~cuda~ and ~nvrtc~ libraries
- zbg v0.3.0: New release 2 years later
- Announcing Raven: Scientific Computing for OCaml (Alpha Release)
- Outreachy Demo Day August 2025
- Other OCaml News
The FreeBSD workers have been getting [slower]( +(https://github.com/ocurrent/opam-repo-ci/issues/449): jobs that should take a few minutes are now timing out after 60 minutes. My first instinct was that ZFS was acting strangely.
+ +I checked the classic ZFS performance indicators:
+ +-
+
- Pool health:
zpool status- ONLINE, no errors
+ - ARC hit ratio:
sysctl kstat.zfs.misc.arcstats.hits kstat.zfs.misc.arcstats.misses- 98.8% (excellent!)
+ - Fragmentation:
zpool list- 53% (high but not catastrophic)
+ - I/O latency:
zpool iostat -v 1 3andiostat -x 1 3- 1ms read/write (actually pretty good)
+
But the sync command was taking 70-160ms when it should be under 10ms for an SSD. We don’t need sync as the disk has disposable CI artefacts, so why not try:
zfs set sync=disabled obuilder
+The sync times improved to 40-50ms, but the CI jobs were still crawling.
+ +I applied some ZFS tuning to try to improve things:
+ +# Crank up those queue depths
+sysctl vfs.zfs.vdev.async_read_max_active=32
+sysctl vfs.zfs.vdev.async_write_max_active=32
+sysctl vfs.zfs.vdev.sync_read_max_active=32
+sysctl vfs.zfs.vdev.sync_write_max_active=32
+
+# Speed up transaction groups
+sysctl vfs.zfs.txg.timeout=1
+sysctl vfs.zfs.dirty_data_max=8589934592
+
+# Optimize for metadata
+zfs set atime=off obuilder
+zfs set primarycache=metadata obuilder
+sysctl vfs.zfs.arc.meta_balance=1000
+However, these changes were making no measurable difference to the actual performance.
+ +For comparison, I ran one of the CI steps on an identical machine, which was running Ubuntu with BTRFS:-
+ +opam install astring.0.8.5 base-bigarray.base base-domains.base base-effects.base base-nnp.base base-threads.base base-unix.base base64.3.5.1 bechamel.0.5.0 camlp-streams.5.0.1 cmdliner.1.3.0 cppo.1.8.0 csexp.1.5.2 dune.3.20.0 either.1.0.0 fmt.0.11.0 gg.1.0.0 jsonm.1.0.2 logs.0.9.0 mdx.2.5.0 ocaml.5.3.0 ocaml-base-compiler.5.3.0 ocaml-compiler.5.3.0 ocaml-config.3 ocaml-options-vanilla.1 ocaml-version.4.0.1 ocamlbuild.0.16.1 ocamlfind.1.9.8 optint.0.3.0 ounit2.2.2.7 re.1.13.2 repr.0.7.0 result.1.5 seq.base stdlib-shims.0.3.0 topkg.1.1.0 uutf.1.0.4 vg.0.9.5
+This took < 3 minutes, but the worker logs showed the same step took 35 minutes. What could cause such a massive difference on identical hardware?
+ +On macOS, I’ve previously seen problems when the number of mounted filesystems got to around 1000. mount would take t minutes to complete. I wondered, how many file systems are mounted?
# mount | grep obuilder | wc -l
+ 33787
+Now, that’s quite a few file systems. Historically, our FreeBSD workers had tiny SSDs, circa 128GB, but with the move to a new server with a 1.7TB SSD disk and using the same 25% prune threshold, the number of mounted file systems has become quite large.
+ +I gradually increased the prune threshold and waited for ocurrent/ocluster to prune jobs. With the threshold at 90% the number of file systems was down to ~5,000, and performance was restored.
+ +It’s not really a bug; it’s just an unexpected side effect of having a large number of mounted file systems. On macOS, the resolution was to unmount all the file systems at the end of each job, but that’s easy when the concurrency is limited to one and more tricky when the concurrency is 20 jobs.
diff --git a/data/planet/tunbury/animating-3d-models-in-ocaml-with-claude.md b/data/planet/tunbury/animating-3d-models-in-ocaml-with-claude.md new file mode 100644 index 0000000000..cef4a767d6 --- /dev/null +++ b/data/planet/tunbury/animating-3d-models-in-ocaml-with-claude.md @@ -0,0 +1,88 @@ +--- +title: Animating 3D models in OCaml with Claude +description: "In the week, Jon mentioned UTM, which uses Apple\u2019s Hypervisor virtualisation + framework to run ARM64 operating systems on Apple Silicon. It looked awesome, and + the speed of virtualised macOS was fantastic. It also offers x86_64 emulation; we + mused how well it would perform running Windows, but found it disappointing." +url: https://www.tunbury.org/2025/06/07/claude-animates-in-ocaml/ +date: 2025-06-07T00:00:00-00:00 +preview_image: https://www.tunbury.org/images/human.png +authors: +- Mark Elvers +source: +ignore: +--- + +In the week, Jon mentioned UTM, which uses Apple’s Hypervisor virtualisation framework to run ARM64 operating systems on Apple Silicon. It looked awesome, and the speed of virtualised macOS was fantastic. It also offers x86_64 emulation; we mused how well it would perform running Windows, but found it disappointing.
+ +I was particularly interested in this because I am stuck in the past with macOS Monterey on my Intel Mac Pro ‘trashcan’, as I have a niche Windows application that I can’t live without. A few years ago, I got a prototype running written in Swift. I never finished it as other events got in the way. The learning curve of SceneKit and Blender was intense. I still had the Collada files on my machine and today, of course, we have Claude.
+ +“How would I animate a Collada (.dae) file using OCaml?”. Claude acknowledged the complexity and proposed that lablgl, the OCaml bindings for OpenGL, would be a good starting point. Claude obliged and wrote the entire pipeline, giving me opam commands and Dune configuration files.
The code wouldn’t build, so I looked for the API for labgl. The library seemed old, with no recent activity. I mentioned this to Claude; he was happy to suggest an alternative approach of tgls, thin OpenGL bindings, with tsdl, SDL2 bindings, or the higher-level API from raylib. The idea of a high-level API sounded better, so I asked Claude to rewrite it with raylib.
The code had some compilation issues. Claude had proposed Mesh.gen_cube, which didn’t exist. Claude consulted the API documentation and found gen_mesh_cube instead. This went through several iterations, with Model.load becoming load_model and Model.draw_ex becoming draw_model_ex, etc. Twenty-two versions later, the code nearly compiles. This block continued to fail with two issues. The first being Array.find doesn’t exist and the second being that the type inferred for a was wrong. There are two types and they both contain target: string;. I manually fixed this with (a:animation_channel) and used match Array.find_opt ... with instead of the try ... with.
(* Update animations *)
+let update_object_animations objects animations elapsed_time =
+ Array.map (fun obj ->
+ try
+ let anim = Array.find (fun a -> a.target = obj.name) animations in
+ (* Loop animation *)
+ let loop_time = mod_float elapsed_time anim.duration in
+ let new_transform = interpolate_animation anim loop_time in
+ { obj with current_transform = new_transform }
+ with
+ Not_found -> obj
+ ) objects
+There were still many unused variables, but the code could be built using dune build --release.
Unfortunately, it couldn’t load my Collada file as the load functions were just stubs! Claude duly obliged and wrote a simple XML parser using regular expressions through the Str library, but interestingly suggested that I include xmlm as a dependency. Adding the parser broke the code, and it no longer compiled. The issue was similar to above; the compiler had inferred a type that wasn’t what Claude expected. I fixed this as above. The code also had some issues with the ordering - functions were used before they were defined. Again, this was an easy fix.
The parser still didn’t work, so I suggested ditching the regular expression-based approach and using xmlm instead. This loaded the mesh; it looked bad, but I could see that it was my mesh. However, it still didn’t animate, and I took a wrong turn here. I told Claude that the Collada file contained both the mesh and the animation, but that’s not right. It has been a while since I created the Collada files, and I had forgotten that the animation and the mesh definitions were in different files.
I asked Claude to improve the parser so that it would expect the animation data to be in the same file as the mesh. This is within the specification for Collada, but this was not the structure of my file.
+ +Is there a better approach than dealing with the complexity of writing a Collada XML parser? What formats are supported by raylib?
In a new thread, I asked, “Using OCaml with Raylib, what format should I use for my 3D mode and animation data?”. Claude suggested GLTF 2.0. As my animation is in Blender, it can be exported in GLTF format. Let’s try it!
+ +Claude used the raylib library to read and display a GLTF file and run the animation. The code was much shorter, but … it didn’t compile. I wrote to Claude, “The API for Raylib appears to be different to the one you have used. For example, camera3d.create doesn’t take named parameters, camera3d.prespective should be cameraprojection.perspective etc.” We set to work, and a dozen versions later, we built it successfully.
It didn’t work, though; the console produced an error over and over:
+ +Joint attribute data format not supported, use vec4 u8
+This looked like a problem with the model. I wondered if my GLTF file was compatible with raylib. I asked Claude if he knew of any validation tools, and he suggested an online viewer. This loaded my file perfectly and animated it in the browser. Claude also gave me some simple code to validate, which only loaded the model.
let main () =
+ init_window 800 600 "Static Model Test";
+ let camera = Camera3D.create
+ (Vector3.create 25.0 25.0 25.0)
+ (Vector3.create 0.0 0.0 0.0)
+ (Vector3.create 0.0 1.0 0.0)
+ 45.0 CameraProjection.Perspective in
+
+ let model = load_model "assets/character.gltf" in
+
+ while not (window_should_close ()) do
+ begin_drawing ();
+ clear_background Color.darkgray;
+ begin_mode_3d camera;
+ draw_model model (Vector3.create 0.0 0.0 0.0) 1.0 Color.white;
+ draw_grid 10 1.0;
+ end_mode_3d ();
+ draw_text "Static Model Test" 10 10 20 Color.white;
+ end_drawing ()
+ done;
+
+ unload_model model;
+ close_window ()
+Even this didn’t work! As I said at the top, it’s been a few years since I looked at this, and I still had Blender installed on my machine: version 2.83.4. The current version is 4.4, so I decided to upgrade. The GLTF export in 4.4 didn’t work on my Mac and instead displayed a page of Python warnings about numpy. On the Blender Forum, this thread showed me how to fix it. Armed with a new GLTF file, the static test worked. Returning to the animation code showed that it worked with the updated file; however, there are some significant visual distortions. These aren’t present when viewed in Blender, which I think comes down to how the library interpolates between keyframes. I will look into this another day.
I enjoyed the collaborative approach. I’m annoyed with myself for not remembering the separate file with the animation data. However, I think the change of direction from Collada to GLTF was a good decision, and the speed at which Claude can explore ideas is very impressive.
diff --git a/data/planet/tunbury/base-images-for-oxcaml.md b/data/planet/tunbury/base-images-for-oxcaml.md new file mode 100644 index 0000000000..65bd2e54ef --- /dev/null +++ b/data/planet/tunbury/base-images-for-oxcaml.md @@ -0,0 +1,64 @@ +--- +title: Base images for OxCaml +description: As @dra27 suggested, I first added support in ocurrent/ocaml-version. + I went with the name flambda2, which matched the name in the opam package. +url: https://www.tunbury.org/2025/06/10/oxcaml-base-images/ +date: 2025-06-10T00:00:00-00:00 +preview_image: https://www.tunbury.org/images/ocaml-logo.png +authors: +- Mark Elvers +source: +ignore: +--- + +As @dra27 suggested, I first added support in ocurrent/ocaml-version. I went with the name flambda2, which matched the name in the opam package.
Wherever I found the type Flambda, I added Flambda2. I added a list of OxCaml versions in the style of the unreleased betas and a function is_oxcaml to test if the variant is of type Flambda2, closely following the is_multicore design! The final change was to additional_packages concatenated ocaml-options-only- to flambda2 - again, this change was also needed for multicore.
It was a relatively minor change to the base-image-builder, adding Ocaml_version.Releases.oxcaml to the available switches on AMD64 and ARM64. Following the precedent set by maybe_add_beta and maybe_add_multicore, I added maybe_add_jst, which added the Jane Street opam repository for these builds.
The builds mostly failed because they depended on autoconf, which isn’t included by default on most distributions. Looking in the dockerfile, there is a function called ocaml_depexts, which includes zstd for OCaml > 5.1.0. I extended this function to include autoconf when building OxCaml.
The Arch Linux builds failed due to missing which, so I added this as I did for autoconf
The following are working:
+ +-
+
- Ubuntu 24.10, 24.04, 22.04 +
- OpenSUSE Tumbleweed +
- Fedora 42, 41 +
- Debian Unstable, Testing, 12 +
- Arch +
Failures
+ +-
+
- Alpine 3.21
+
-
+
- missing
linux/auxvec.hheader
+
+ - missing
- OpenSUSE 15.6
+
-
+
- autoconf is too old in the distribution +
+ - Debian 11
+
-
+
- autoconf is too old in the distribution +
+ - Oracle Linux 9, 8
+
-
+
- autoconf is too old in the distribution +
+
There is some discussion about whether building these with the base image builder is the best approach, so I won’t create PRs at this time. My branches are:
+-
+
- https://github.com/mtelvers/ocaml-version.git +
- https://github.com/mtelvers/ocaml-dockerfile.git#oxcaml +
- https://github.com/mtelvers/docker-base-images#oxcaml +
Equinix has stopped commercial sales of Metal and will sunset the service at the end of June 2026. Equinix have long been a supporter of OCaml and has provided free credits to use on their Metal platform. These credits are coming to an end at the end of this month, meaning that we need to move some of our services away from Equinix. We have two new four-node blade servers, which will become the new home for these services. The blades have dual 10C/20T processors with either 192GB or 256GB of RAM and a combination of SSD and spinning disk.
+ +192GB, 20C/40T with 1.1TB SSD, 2 x 6T disks
+-
+
- rosemary: FreeBSD CI Worker (releasing spring & summer) +
- oregano: OpenBSD CI Worker (releasing bremusa) +
- basil: docs-ci (new implementation, eventually replacing eumache) +
- mint: spare +
256GB, 20C/40T with 1.5TB SSD, 2 x 8T disks
+-
+
- thyme: Equinix c2-2 (registry.ci.dev) +
- chives: Equinix c2-4 (opam-repo-ci) + Equinix c2-3 (OCaml-ci) + Equinix c2-1 (preview.dune.dev) +
256GB, 20C/40T with 1.1TB SSD, 2 x 6T disks
+-
+
- dill: spare +
- sage: spare +
VMs currently running on hopi can be redeployed to chives, allowing hopi to be redeployed.
+ +Machines which can then be recycled are:
+-
+
- sleepy (4C) +
- grumpy (4C) +
- doc (4C) +
- spring (8T) +
- tigger +
- armyofdockerness +
We have changed our mind about using dm-cache in the SSD/RAID1 configuration. The current thinking is that the mechanical drives would be better served as extra capacity for our distributed ZFS infrastructure, where we intend to have two copies of all data, and these disks represent ~100TB of storage.
As mentioned previously, we have a deadline of Wednesday, 30th April, to move the workloads from the Equinix machines or incur hosting fees.
+ +I also noted that the SSD capacity is 1.7TB in all cases. The new distribution is:
+ +-
+
- rosemary: FreeBSD CI Worker (releasing spring & summer) +
- oregano: OpenBSD CI Worker (releasing bremusa) +
- basil: Equinix c2-2 (registry.ci.dev) +
- mint: @mte24 workstation +
- thyme: spare +
- chives: Equinix c2-4 (opam-repo-ci) + Equinix c2-3 (OCaml-ci) + Equinix c2-1 (preview.dune.dev) +
- dill: spare +
- sage: docs-ci (new implementation, eventually replacing eumache) +
Our PeerTube installation at watch.ocaml.org holds hundreds of videos we wouldn’t want to lose! It’s a VM hosted at Scaleway so the chances of a loss are pretty small, but having a second copy would give us extra reassurance. I’m going to use Borg Backup.
+ +Here’s the list of features (taken directly from their website):
+ +-
+
- Space-efficient storage of backups. +
- Secure, authenticated encryption. +
- Compression: lz4, zstd, zlib, lzma or none. +
- Mountable backups with FUSE. +
- Easy installation on multiple platforms: Linux, macOS, BSD, … +
- Free software (BSD license). +
- Backed by a large and active open source community. +
We have several OBuilder workers with one or more unused hard disks, which would make ideal backup targets.
+ +In this case, I will format and mount sdc as /home on one of the workers.
parted /dev/sdc mklabel gpt
+parted /dev/sdc mkpart primary ext4 0% 100%
+mkfs.ext4 /dev/sdc1
+Add this to /etc/fstab and run mount -a.
/dev/sdc1 /home ext4 defaults 0 2
+Create a user borg.
adduser --disabled-password --gecos '@borg' --home /home/borg borg
+On both machines, install the application borg.
apt install borgbackup
+On the machine we want to backup, generate an SSH key and copy it to the authorized_keys file for user borg on the target server. Ensure that chmod and chown are correct.
ssh-keygen -t ed25519 -f ~/.ssh/borg_backup_key
+Add lines to the .ssh/config for ease of connection. We can now ssh backup-server without any prompts.
Host backup-server
+ HostName your.backup.server.com
+ User borg
+ IdentityFile ~/.ssh/borg_backup_key
+ ServerAliveInterval 60
+ ServerAliveCountMax 3
+Borg supports encrypting the backup at rest on the target machine. The data is publicly available in this case, so encryption seems unnecessary.
+ +On the machine to be backed up, run.
+ +borg init --encryption=none backup-server:repo
+We can now perform a backup or two and see how the deduplication works.
+ +# borg create backup-server:repo::test /var/lib/docker/volumes/postgres --compression lz4 --stats --progress
+------------------------------------------------------------------------------
+Repository: ssh://backup-server/./repo
+Archive name: test
+Archive fingerprint: 627242cb5b65efa23672db317b4cdc8617a78de4d8e195cdd1e1358ed02dd937
+Time (start): Sat, 2025-06-14 13:32:27
+Time (end): Sat, 2025-06-14 13:32:38
+Duration: 11.03 seconds
+Number of files: 3497
+Utilization of max. archive size: 0%
+------------------------------------------------------------------------------
+ Original size Compressed size Deduplicated size
+This archive: 334.14 MB 136.28 MB 132.79 MB
+All archives: 334.14 MB 136.28 MB 132.92 MB
+
+ Unique chunks Total chunks
+Chunk index: 942 1568
+------------------------------------------------------------------------------
+# borg create backup-server:repo::test2 /var/lib/docker/volumes/postgres --compression lz4 --stats --progress
+------------------------------------------------------------------------------
+Repository: ssh://backup-server/./repo
+Archive name: test2
+Archive fingerprint: 572bf2225b3ab19afd32d44f058a49dc2b02cb70c8833fa0b2a1fb5b95526bff
+Time (start): Sat, 2025-06-14 13:33:05
+Time (end): Sat, 2025-06-14 13:33:06
+Duration: 1.43 seconds
+Number of files: 3497
+Utilization of max. archive size: 0%
+------------------------------------------------------------------------------
+ Original size Compressed size Deduplicated size
+This archive: 334.14 MB 136.28 MB 9.58 MB
+All archives: 668.28 MB 272.55 MB 142.61 MB
+
+ Unique chunks Total chunks
+Chunk index: 971 3136
+------------------------------------------------------------------------------
+# borg list backup-server:repo
+test Sat, 2025-06-14 13:32:27 [627242cb5b65efa23672db317b4cdc8617a78de4d8e195cdd1e1358ed02dd937]
+test2 Sat, 2025-06-14 13:33:05 [572bf2225b3ab19afd32d44f058a49dc2b02cb70c8833fa0b2a1fb5b95526bff]
+Let’s run this every day via by placing a script borgbackup in /etc/cron.daily. The paths given are just examples…
#!/bin/bash
+
+# Configuration
+REPOSITORY="backup-server:repo"
+
+# What to backup
+BACKUP_PATHS="
+/home
+"
+
+# What to exclude
+EXCLUDE_ARGS="
+--exclude '*.tmp'
+--exclude '*.log'
+"
+
+# Logging function
+log() {
+ logger -t "borg-backup" "$1"
+ echo "$(date '+%Y-%m-%d %H:%M:%S') - $1"
+}
+
+log "========================================"
+log "Starting Borg backup"
+
+# Check if borg is installed
+if ! command -v borg &> /dev/null; then
+ log "ERROR: borg command not found"
+ exit 1
+fi
+
+# Test repository access
+if ! borg info "$REPOSITORY" &> /dev/null; then
+ log "ERROR: Cannot access repository $REPOSITORY"
+ log "Make sure repository exists and SSH key is set up"
+ exit 1
+fi
+
+# Create backup
+log "Creating backup archive..."
+if borg create \
+ "$REPOSITORY::backup-{now}" \
+ $BACKUP_PATHS \
+ $EXCLUDE_ARGS \
+ --compression lz4 \
+ --stats 2>&1 | logger -t "borg-backup"; then
+ log "Backup created successfully"
+else
+ log "ERROR: Backup creation failed"
+ exit 1
+fi
+
+# Prune old backups
+log "Pruning old backups..."
+if borg prune "$REPOSITORY" \
+ --keep-daily=7 \
+ --keep-weekly=4 \
+ --keep-monthly=6 \
+ --stats 2>&1 | logger -t "borg-backup"; then
+ log "Pruning completed successfully"
+else
+ log "WARNING: Pruning failed, but backup was successful"
+fi
+
+# Monthly repository check (on the 1st of each month)
+if [ "$(date +%d)" = "01" ]; then
+ log "Running monthly repository check..."
+ if borg check "$REPOSITORY" 2>&1 | logger -t "borg-backup"; then
+ log "Repository check passed"
+ else
+ log "WARNING: Repository check failed"
+ fi
+fi
+
+log "Backup completed successfully"
+log "========================================"
+Check the logs…
+ +journalctl -t borg-backup
+Over the weekend, I decided to extend my Box tool to incorporate file upload. There is a straightforward POST API for this with a curl one-liner given in the Box documentation. Easy.
The documentation for Cohttp-eio.Client only gives the function signature for post, but it looked pretty similar to get, which I had already been working with. The README for Cohttp gave me pause when I read this comment about multipart forms.
++ +Multipart form data is not supported out of the box but is provided by external libraries
+
Of the three options given, the second option looked abandoned, while the third said it didn’t support streaming, so I went with the first one dionsaure/multipart_form.
+ +The landing page included an example encoder. A couple of external functions are mentioned, and I found example code for these in test/test.ml. This built, but didn’t work against Box. I ran nc -l 127.0.0.1 6789 and set that as the API endpoint for both the curl and my application. This showed I was missing the Content-Type header in the part boundary. It should be application/octet-stream.
There is a ~header parameter to part, and I hoped for a Header.add like the Cohttp, but sadly not. See the documentation. There is Header.content_type, but that returns the content type. How do you make it? Header.of_list requires a Field.field list.
In a bit of frustration, I decided to ask Claude. I’ve not tried it before, but I’ve seen some impressive demonstrations. My first lesson here was to be specific. Claude is not a mind reader. After a few questions, I got to this:
+ +Field.(make Content_type.name (Content_type.v `Application `Octet_stream));
+I can see why this was suggested as Content_disposition.v exists, but Content_type.v does not, nor does Field.make. Claude quickly obliged with a new version when I pointed this out but added the Content_type to the HTTP header rather than the boundary header. This went back and forth for a while, with Claude repeatedly suggesting functions which did not exist. I gave up.
On OCaml.org, the multipart-form documentation includes a Used by section that listed dream as the only (external) application which used the library. From the source, I could see Field.Field (field_name, Field.Content_type, v), which looked good.
There is a function Content_type.of_string. I used :MerlinLocate to find the source, which turned out to be an Angstrom parser which returns a Content_type.t. This led me to Content_type.make, and ultimately, I was able to write these two lines:
let v = Content_type.make `Application (`Iana_token "octet-stream") Content_type.Parameters.empty
+let p0 = part ~header:(Header.of_list [ Field (Field_name.content_type, Content_type, v) ]) ...
+As a relatively new adopter of OCaml as my language of choice, the most significant challenge I face is documentation, particularly when I find a library on opam which I want to use. I find this an interesting contrast to the others in the community, where it is often cited that tooling is the most significant barrier to adoption. In my opinion, the time taken to set up a build environment is dwarfed by the time spent in that environment iterating code.
+ +I would like to take this opportunity to thank all contributors to opam repository for their time and effort in making packages available. This post mentions specific packages but only to illustrate my point.
diff --git a/data/planet/tunbury/box-diff-tool.md b/data/planet/tunbury/box-diff-tool.md new file mode 100644 index 0000000000..3a6ef656c1 --- /dev/null +++ b/data/planet/tunbury/box-diff-tool.md @@ -0,0 +1,21 @@ +--- +title: Box Diff Tool +description: Over the weekend, I extended mtelvers/ocaml-box-diff to include the ability + to upload files over 50MB. This is a more complex API which requires a call to https://upload.box.com/api/2.0/files/upload_sessions + by posting JSON containing the name of the file, the folder ID and the file size. + Box replies with various session endpoints which give the URIs to use to upload + the parts and to commit the the file. Box also specifies the size of each part. +url: https://www.tunbury.org/2025/04/12/box-diff/ +date: 2025-04-12T00:00:00-00:00 +preview_image: https://www.tunbury.org/images/box-logo.png +authors: +- Mark Elvers +source: +ignore: +--- + +Over the weekend, I extended mtelvers/ocaml-box-diff to include the ability to upload files over 50MB. This is a more complex API which requires a call to https://upload.box.com/api/2.0/files/upload_sessions by posting JSON containing the name of the file, the folder ID and the file size. Box replies with various session endpoints which give the URIs to use to upload the parts and to commit the the file. Box also specifies the size of each part.
+ +Each part is uploaded with an HTTP PUT of the binary data, with header fields giving the byte range within the overall file along with the SHA for this chunk. Box replies with a part identifier. Once all the parts have been uploaded, an HTTP POST is required to the commit URI, passing a JSON array of all the parts as well as the overall SHA for the file.
+ +I was pleased to be able to reuse stream_of_file, which was written for the small file upload. Additionally, I was able to keep a running total SHA for the data uploaded so far using Sha1.update_string ctx chunk, meaning that I did not need to recompute the overall file SHA at the end.
Everything was going fine until I ran out of disk space. My NVMe, C: drive, is only 256GB, but I have a large, 1.7TB SSD available as D:. How trivial, change a few paths and carry on, but it wasn’t that simple, or was it?
Distilling the problem down to the minimum and excluding all code written by me, the following command fails, but changing src=d:\cache\opam to src=c:\cache\opam works. It’s not the content, as it’s just an empty folder.
ctr run --rm --cni -user ContainerAdministrator -mount type=bind,src=d:\cache\opam,dst=c:\Users\ContainerAdministrator\AppData\Local\opam mcr.microsoft.com/windows/servercore:ltsc2022 my-container cmd /c "curl.exe -L -o c:\Windows\opam.exe https://github.com/ocaml/opam/releases/download/2.3.0/opam-2.3.0-x86_64-windows.exe && opam.exe init --debug-level=3 -y"
+The failure point is the ability to create the lock file config.lock. Checking the code, the log entry is written before the lock is acquired. If c:\Users\ContainerAdministrator\AppData\Local\opam is not a bind mount, or the bind mount is on C:, then it works.
01:26.722 CLIENT updating repository state
+01:26.722 GSTATE LOAD-GLOBAL-STATE @ C:\Users\ContainerAdministrator\AppData\Local\opam
+01:26.723 SYSTEM LOCK C:\Users\ContainerAdministrator\AppData\Local\opam\lock (none => read)
+01:26.723 SYSTEM LOCK C:\Users\ContainerAdministrator\AppData\Local\opam\config.lock (none => write)
+Suffice it to say, I spent a long time trying to resolve this. I’ll mention a couple of interesting points that appeared along the way. Firstly, files created on D: effectively appear as hard links, and the Update Sequence Number, USN, is 0.
C:\> fsutil file layout d:\cache\opam\lock
+
+********* File 0x000400000001d251 *********
+File reference number : 0x000400000001d251
+File attributes : 0x00000020: Archive
+File entry flags : 0x00000000
+Link (ParentID: Name) : 0x000c00000000002d: HLINK Name : \cache\opam\lock
+...
+LastUsn : 0
+...
+The reason behind this is down to Windows defaults:
+ +-
+
- Windows still likes to create the legacy 8.3 MS-DOS file names on the system volume,
C:, which explains the difference betweenHLINKandNTFS+DOS. Runningfsutil 8dot3name set d: 0will enable the creation of the old-style file names.
+ - Drive
C:has a USN journal created automatically, as it’s required for Windows to operate, but it isn’t created by default on other drives. Runningfsutil usn createjournal d: m=32000000 a=8000000will create the journal.
+
C:\> fsutil file layout c:\cache\opam\lock
+
+********* File 0x000300000002f382 *********
+File reference number : 0x000300000002f382
+File attributes : 0x00000020: Archive
+File entry flags : 0x00000000
+Link (ParentID: Name) : 0x000b0000000271d1: NTFS+DOS Name: \cache\opam\lock
+...
+LastUsn : 16,897,595,224
+...
+Sadly, neither of these insights makes any difference to my problem. I did notice that containerd 2.1.3 had been released, where I had been using 2.1.1. Upgrading didn’t fix the issue, but it did affect how the network namespaces were created. More later.
I decided to both ignore the problem and try it on another machine. After all, this problem was only a problem because my C: was too small. I created a QEMU VM with a 40GB C: and a 1TB D: and installed everything, and it worked fine with the bind mount on D: even without any of the above tuning and even with D: formatted using ReFS, rather than NTFS.
Trying on another physical machine with a single large spinning disk as C: also worked as anticipated.
In both of these new installations, I used containerd 2.1.3 and noticed that the behaviour I had come to rely upon seemed to have changed. If you recall, in this post, I found the network namespace GUID by running ctr run on a standard Windows container and then ctr container info in another window. This no longer worked reliably, as the namespace was removed when the container exited. Perhaps it always should have been?
I need to find out how to create these namespaces. PowerShell has a cmdlet Get-HnsNetwork, but none of the GUID values there match the currently running namespaces I observe from ctr container info. The source code of containerd is on GitHub..
When you pass --cni to the ctr command, it populates the network namespace from NetNewNS. Snippet from cmd/ctr/commands/run/run_windows.go
if cliContext.Bool("cni") {
+ ns, err := netns.NewNetNS("")
+ if err != nil {
+ return nil, err
+ }
+ opts = append(opts, oci.WithWindowsNetworkNamespace(ns.GetPath()))
+ }
+NewNetNS is defined in pkg/netns/netns_windows.go
// NetNS holds network namespace for sandbox
+type NetNS struct {
+ path string
+}
+
+// NewNetNS creates a network namespace for the sandbox.
+func NewNetNS(baseDir string) (*NetNS, error) {
+ temp := hcn.HostComputeNamespace{}
+ hcnNamespace, err := temp.Create()
+ if err != nil {
+ return nil, err
+ }
+
+ return &NetNS{path: hcnNamespace.Id}, nil
+}
+Following the thread, and cutting out a few steps in the interest of brevity, we end up in vendor/github.com/Microsoft/hcsshim/hcn/zsyscall_windows.go which calls a Win32 API.
func _hcnCreateNamespace(id *_guid, settings *uint16, namespace *hcnNamespace, result **uint16) (hr error) {
+ hr = procHcnCreateNamespace.Find()
+ if hr != nil {
+ return
+ }
+ r0, _, _ := syscall.SyscallN(procHcnCreateNamespace.Addr(), uintptr(unsafe.Pointer(id)), uintptr(unsafe.Pointer(settings)), uintptr(unsafe.Pointer(namespace)), uintptr(unsafe.Pointer(result)))
+ if int32(r0) < 0 {
+ if r0&0x1fff0000 == 0x00070000 {
+ r0 &= 0xffff
+ }
+ hr = syscall.Errno(r0)
+ }
+ return
+}
+PowerShell provides Get-HnsNamespace to list available namespaces. These are the droids values I’ve been looking for to put in config.json! However, by default there are no cmdlets to create them. The installation PowerShell script for containerd pulls in hns.psm1 for containerd, has a lot of interesting cmdlets, such as New-HnsNetwork, but not a cmdlet to create a namespace. There is also hns.v2.psm1, which does have New-HnsNamespace.
PS C:\Users\Administrator> curl.exe -o hns.v2.psm1 -L https://raw.githubusercontent.com/microsoft/SDN/refs/heads/master/Kubernetes/windows/hns.v2.psm1
+ % Total % Received % Xferd Average Speed Time Time Time Current
+ Dload Upload Total Spent Left Speed
+100 89329 100 89329 0 0 349k 0 --:--:-- --:--:-- --:--:-- 353k
+
+PS C:\Users\Administrator> Import-Module .\hns.v2.psm1
+WARNING: The names of some imported commands from the module 'hns.v2' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
+
+PS C:\Users\Administrator> New-HnsNamespace
+HcnCreateNamespace -- HRESULT: 2151350299. Result: {"Success":false,"Error":"Invalid JSON document string. {{CreateWithCompartment,UnknownField}}","ErrorCode":2151350299}
+At C:\Users\Administrator\hns.v2.psm1:2392 char:13
++ throw $errString
++ ~~~~~~~~~~~~~~~~
+ + CategoryInfo : OperationStopped: (HcnCreateNamesp...de":2151350299}:String) [], RuntimeException
+ + FullyQualifiedErrorId : HcnCreateNamespace -- HRESULT: 2151350299. Result: {"Success":false,"Error":"Invalid JSON document string. {{CreateWithCompartment,UnknownField}}","ErrorCode":2151350299}
+With a lot of frustration, I decided to have a go at calling the Win32 API from OCaml. This resulted in mtelvers/hcn-namespace, which allows me to create the namespaces by running hcn-namespace create. These namespaces appear in the output from Get-HnsNamespace and work correctly in config.json.
Run hcn-namespace.exe create, and then populate "networkNamespace": "<GUID>" with the GUID provided and run with ctr run --rm -cni --config config.json.
The log from an OBuilder job starts with the steps needed to reproduce the job locally. This boilerplate output assumes that all OBuilder jobs start from a Docker base image, but on some operating systems, such as FreeBSD and macOS, OBuilder uses ZFS base images. On OpenBSD and Windows, it uses QEMU images. The situation is further complicated when the issue only affects a specific architecture that may be unavailable to the user.
+ +2025-05-08 13:29.37: New job: build bitwuzla-cxx.0.7.0, using opam 2.3
+ from https://github.com/ocaml/opam-repository.git#refs/pull/27768/head (55a47416d532dc829d9111297970934a21a1b1c4)
+ on macos-homebrew-ocaml-4.14/amd64
+
+To reproduce locally:
+
+cd $(mktemp -d)
+git clone --recursive "https://github.com/ocaml/opam-repository.git" && cd "opam-repository" && git fetch origin "refs/pull/27768/head" && git reset --hard 55a47416
+git fetch origin master
+git merge --no-edit b8a7f49af3f606bf8a22869a1b52b250dd90092e
+cat > ../Dockerfile <<'END-OF-DOCKERFILE'
+
+FROM macos-homebrew-ocaml-4.14
+USER 1000:1000
+RUN ln -f ~/local/bin/opam-2.3 ~/local/bin/opam
+RUN opam init --reinit -ni
+RUN opam option solver=builtin-0install && opam config report
+ENV OPAMDOWNLOADJOBS="1"
+ENV OPAMERRLOGLEN="0"
+ENV OPAMPRECISETRACKING="1"
+ENV CI="true"
+ENV OPAM_REPO_CI="true"
+RUN rm -rf opam-repository/
+COPY --chown=1000:1000 . opam-repository/
+RUN opam repository set-url -k local --strict default opam-repository/
+RUN opam update --depexts || true
+RUN opam pin add -k version -yn bitwuzla-cxx.0.7.0 0.7.0
+RUN opam reinstall bitwuzla-cxx.0.7.0; \
+ res=$?; \
+ test "$res" != 31 && exit "$res"; \
+ export OPAMCLI=2.0; \
+ build_dir=$(opam var prefix)/.opam-switch/build; \
+ failed=$(ls "$build_dir"); \
+ partial_fails=""; \
+ for pkg in $failed; do \
+ if opam show -f x-ci-accept-failures: "$pkg" | grep -qF "\"macos-homebrew\""; then \
+ echo "A package failed and has been disabled for CI using the 'x-ci-accept-failures' field."; \
+ fi; \
+ test "$pkg" != 'bitwuzla-cxx.0.7.0' && partial_fails="$partial_fails $pkg"; \
+ done; \
+ test "${partial_fails}" != "" && echo "opam-repo-ci detected dependencies failing: ${partial_fails}"; \
+ exit 1
+
+
+END-OF-DOCKERFILE
+docker build -f ../Dockerfile .
+It is, therefore, difficult to diagnose the issue on these operating systems and on esoteric architectures. Is it an issue with the CI system or the job itself?
+ +My approach is to get myself into an interactive shell at the point in the build where the failure occurs. On Linux and FreeBSD, the log is available in /var/log/syslog or /var/log/messages respectively. On macOS, this log is written to ocluster.log. macOS workers are single-threaded, so the worker must be paused before progressing.
Each step in an OBuilder job consists of taking a snapshot of the previous layer, running a command in that layer, and keeping or discarding the layer depending on the command’s success or failure. On macOS, layers are ZFS snapshots mounted over the Homebrew directory and the CI users’ home directory. We can extract the appropriate command from the logs.
+ +2025-05-08 14:31.17 application [INFO] Exec "zfs" "clone" "-o" "canmount=noauto" "--" "obuilder/result/a67e6d3b460fa52b5c57581e7c01fa74ddca0a0b5462fef34103a09e87f3feec@snap" "obuilder/result/af09425cd7744c7b32ed000b11db90295142f3d3430fddb594932d5c02343b40"
+2025-05-08 14:31.17 application [INFO] Exec "zfs" "mount" "obuilder/result/af09425cd7744c7b32ed000b11db90295142f3d3430fddb594932d5c02343b40"
+2025-05-08 14:31.17 application [INFO] Exec "zfs" "clone" "-o" "mountpoint=none" "--" "obuilder/result/a67e6d3b460fa52b5c57581e7c01fa74ddca0a0b5462fef34103a09e87f3feec/brew@snap" "obuilder/result/af09425cd7744c7b32ed000b11db90295142f3d3430fddb594932d5c02343b40/brew"
+2025-05-08 14:31.17 application [INFO] Exec "zfs" "clone" "-o" "mountpoint=none" "--" "obuilder/result/a67e6d3b460fa52b5c57581e7c01fa74ddca0a0b5462fef34103a09e87f3feec/home@snap" "obuilder/result/af09425cd7744c7b32ed000b11db90295142f3d3430fddb594932d5c02343b40/home"
+cannot open 'obuilder/result/af09425cd7744c7b32ed000b11db90295142f3d3430fddb594932d5c02343b40@snap': dataset does not exist
+2025-05-08 14:31.17 application [INFO] Exec "zfs" "clone" "--" "obuilder/cache/c-opam-archives@snap" "obuilder/cache-tmp/8608-c-opam-archives"
+2025-05-08 14:31.17 application [INFO] Exec "zfs" "clone" "--" "obuilder/cache/c-homebrew@snap" "obuilder/cache-tmp/8609-c-homebrew"
+2025-05-08 14:31.18 obuilder [INFO] result_tmp = /Volumes/obuilder/result/af09425cd7744c7b32ed000b11db90295142f3d3430fddb594932d5c02343b40
+2025-05-08 14:31.18 application [INFO] Exec "zfs" "set" "mountpoint=/Users/mac1000" "obuilder/result/af09425cd7744c7b32ed000b11db90295142f3d3430fddb594932d5c02343b40/home"
+2025-05-08 14:31.18 application [INFO] Exec "zfs" "set" "mountpoint=/usr/local" "obuilder/result/af09425cd7744c7b32ed000b11db90295142f3d3430fddb594932d5c02343b40/brew"
+2025-05-08 14:31.18 obuilder [INFO] src = /Volumes/obuilder/cache-tmp/8608-c-opam-archives, dst = /Users/mac1000/.opam/download-cache, type rw
+2025-05-08 14:31.18 application [INFO] Exec "zfs" "set" "mountpoint=/Users/mac1000/.opam/download-cache" "obuilder/cache-tmp/8608-c-opam-archives"
+Unmount successful for /Volumes/obuilder/cache-tmp/8608-c-opam-archives
+2025-05-08 14:31.18 obuilder [INFO] src = /Volumes/obuilder/cache-tmp/8609-c-homebrew, dst = /Users/mac1000/Library/Caches/Homebrew, type rw
+2025-05-08 14:31.18 application [INFO] Exec "zfs" "set" "mountpoint=/Users/mac1000/Library/Caches/Homebrew" "obuilder/cache-tmp/8609-c-homebrew"
+Unmount successful for /Volumes/obuilder/cache-tmp/8609-c-homebrew
+2025-05-08 14:31.19 application [INFO] Exec "sudo" "dscl" "." "list" "/Users"
+2025-05-08 14:31.19 application [INFO] Exec "sudo" "-u" "mac1000" "-i" "getconf" "DARWIN_USER_TEMP_DIR"
+2025-05-08 14:31.19 application [INFO] Fork exec "sudo" "su" "-l" "mac1000" "-c" "--" "source ~/.obuilder_profile.sh && env 'TMPDIR=/var/folders/s_/z7_t3bvn5txfn81hk9p3ntfw0000z8/T/' 'OPAM_REPO_CI=true' 'CI=true' 'OPAMPRECISETRACKING=1' 'OPAMERRLOGLEN=0' 'OPAMDOWNLOADJOBS=1' "$0" "$@"" "/usr/bin/env" "bash" "-c" "opam reinstall bitwuzla-cxx.0.7.0;
+ res=$?;
+ test "$res" != 31 && exit "$res";
+ export OPAMCLI=2.0;
+ build_dir=$(opam var prefix)/.opam-switch/build;
+ failed=$(ls "$build_dir");
+ partial_fails="";
+ for pkg in $failed; do
+ if opam show -f x-ci-accept-failures: "$pkg" | grep -qF "\"macos-homebrew\""; then
+ echo "A package failed and has been disabled for CI using the 'x-ci-accept-failures' field.";
+ fi;
+ test "$pkg" != 'bitwuzla-cxx.0.7.0' && partial_fails="$partial_fails $pkg";
+ done;
+ test "${partial_fails}" != "" && echo "opam-repo-ci detected dependencies failing: ${partial_fails}”;
+ exit 1"
+2025-05-08 14:31.28 worker [INFO] OBuilder partition: 27% free, 2081 items
+2025-05-08 14:31.58 worker [INFO] OBuilder partition: 27% free, 2081 items
+2025-05-08 14:32.28 worker [INFO] OBuilder partition: 27% free, 2081 items
+2025-05-08 14:32.43 application [INFO] Exec "zfs" "inherit" "mountpoint" "obuilder/cache-tmp/8608-c-opam-archives"
+Unmount successful for /Users/mac1000/.opam/download-cache
+2025-05-08 14:32.44 application [INFO] Exec "zfs" "inherit" "mountpoint" "obuilder/cache-tmp/8609-c-homebrew"
+Unmount successful for /Users/mac1000/Library/Caches/Homebrew
+2025-05-08 14:32.45 application [INFO] Exec "zfs" "set" "mountpoint=none" "obuilder/result/af09425cd7744c7b32ed000b11db90295142f3d3430fddb594932d5c02343b40/home"
+Unmount successful for /Users/mac1000
+2025-05-08 14:32.45 application [INFO] Exec "zfs" "set" "mountpoint=none" "obuilder/result/af09425cd7744c7b32ed000b11db90295142f3d3430fddb594932d5c02343b40/brew"
+Unmount successful for /usr/local
+2025-05-08 14:32.46 application [INFO] Exec "zfs" "rename" "--" "obuilder/cache/c-homebrew" "obuilder/cache-tmp/8610-c-homebrew"
+Unmount successful for /Volumes/obuilder/cache/c-homebrew
+2025-05-08 14:32.46 application [INFO] Exec "zfs" "promote" "obuilder/cache-tmp/8609-c-homebrew"
+2025-05-08 14:32.46 application [INFO] Exec "zfs" "destroy" "-f" "--" "obuilder/cache-tmp/8610-c-homebrew"
+Unmount successful for /Volumes/obuilder/cache-tmp/8610-c-homebrew
+2025-05-08 14:32.48 application [INFO] Exec "zfs" "rename" "--" "obuilder/cache-tmp/8609-c-homebrew@snap" "obuilder/cache-tmp/8609-c-homebrew@old-2152"
+2025-05-08 14:32.48 application [INFO] Exec "zfs" "destroy" "-d" "--" "obuilder/cache-tmp/8609-c-homebrew@old-2152"
+2025-05-08 14:32.48 application [INFO] Exec "zfs" "snapshot" "-r" "--" "obuilder/cache-tmp/8609-c-homebrew@snap"
+2025-05-08 14:32.48 application [INFO] Exec "zfs" "rename" "--" "obuilder/cache-tmp/8609-c-homebrew" "obuilder/cache/c-homebrew"
+Unmount successful for /Volumes/obuilder/cache-tmp/8609-c-homebrew
+2025-05-08 14:32.49 application [INFO] Exec "zfs" "rename" "--" "obuilder/cache/c-opam-archives" "obuilder/cache-tmp/8611-c-opam-archives"
+Unmount successful for /Volumes/obuilder/cache/c-opam-archives
+2025-05-08 14:32.50 application [INFO] Exec "zfs" "promote" "obuilder/cache-tmp/8608-c-opam-archives"
+2025-05-08 14:32.50 application [INFO] Exec "zfs" "destroy" "-f" "--" "obuilder/cache-tmp/8611-c-opam-archives"
+Unmount successful for /Volumes/obuilder/cache-tmp/8611-c-opam-archives
+2025-05-08 14:32.51 application [INFO] Exec "zfs" "rename" "--" "obuilder/cache-tmp/8608-c-opam-archives@snap" "obuilder/cache-tmp/8608-c-opam-archives@old-2152"
+2025-05-08 14:32.51 application [INFO] Exec "zfs" "destroy" "-d" "--" "obuilder/cache-tmp/8608-c-opam-archives@old-2152"
+2025-05-08 14:32.51 application [INFO] Exec "zfs" "snapshot" "-r" "--" "obuilder/cache-tmp/8608-c-opam-archives@snap"
+2025-05-08 14:32.52 application [INFO] Exec "zfs" "rename" "--" "obuilder/cache-tmp/8608-c-opam-archives" "obuilder/cache/c-opam-archives"
+Unmount successful for /Volumes/obuilder/cache-tmp/8608-c-opam-archives
+2025-05-08 14:32.52 application [INFO] Exec "zfs" "destroy" "-r" "-f" "--" "obuilder/result/af09425cd7744c7b32ed000b11db90295142f3d3430fddb594932d5c02343b40"
+Unmount successful for /Volumes/obuilder/result/af09425cd7744c7b32ed000b11db90295142f3d3430fddb594932d5c02343b40
+2025-05-08 14:32.58 worker [INFO] OBuilder partition: 27% free, 2081 items
+2025-05-08 14:33.04 worker [INFO] Job failed: "/usr/bin/env" "bash" "-c" "opam reinstall bitwuzla-cxx.0.7.0;
+ res=$?;
+ test "$res" != 31 && exit "$res";
+ export OPAMCLI=2.0;
+ build_dir=$(opam var prefix)/.opam-switch/build;
+ failed=$(ls "$build_dir");
+ partial_fails="";
+ for pkg in $failed; do
+ if opam show -f x-ci-accept-failures: "$pkg" | grep -qF "\"macos-homebrew\""; then
+ echo "A package failed and has been disabled for CI using the 'x-ci-accept-failures' field.";
+ fi;
+ test "$pkg" != 'bitwuzla-cxx.0.7.0' && partial_fails="$partial_fails $pkg";
+ done;
+ test "${partial_fails}" != "" && echo "opam-repo-ci detected dependencies failing: ${partial_fails}";
+ exit 1" failed with exit status 1
+
+Run each of the Exec commands at the command prompt up to the Fork exec. We do need to run it, but we want an interactive shell, so let’s change the final part of the command to bash:
sudo su -l mac1000 -c -- "source ~/.obuilder_profile.sh && env 'TMPDIR=/var/folders/s_/z7_t3bvn5txfn81hk9p3ntfw0000z8/T/' 'OPAM_REPO_CI=true' 'CI=true' 'OPAMPRECISETRACKING=1' 'OPAMERRLOGLEN=0' 'OPAMDOWNLOADJOBS=1' bash"
+Now, at the shell prompt, we can try opam reinstall bitwuzla-cxx.0.7.0. Hopefully, this fails, which proves we have successfully recreated the environment!
$ opam source bitwuzla-cxx.0.7.0
+$ cd bitwuzla-cxx.0.7.0
+$ dune build
+File "vendor/dune", lines 201-218, characters 0-436:
+201 | (rule
+202 | (deps
+203 | (source_tree bitwuzla)
+.....
+216 | %{p0002}
+217 | (run patch -p1 --directory bitwuzla))
+218 | (write-file %{target} "")))))
+(cd _build/default/vendor && /usr/bin/patch -p1 --directory bitwuzla) < _build/default/vendor/patch/0001-api-Add-hook-for-ocaml-z-value.patch
+patching file 'include/bitwuzla/cpp/bitwuzla.h'
+Can't create '/var/folders/s_/z7_t3bvn5txfn81hk9p3ntfw0000z8/T/build_9012b8_dune/patchoEyVbKAjSTw', output is in '/var/folders/s_/z7_t3bvn5txfn81hk9p3ntfw0000z8/T/build_9012b8_dune/patchoEyVbKAjSTw': Permission denied
+patch: **** can't create '/var/folders/s_/z7_t3bvn5txfn81hk9p3ntfw0000z8/T/build_9012b8_dune/patchoEyVbKAjSTw': Permission denied
+This matches the output we see on the CI logs. /var/folders/s_/z7_t3bvn5txfn81hk9p3ntfw0000z8/T is the TMPDIR value set in the environment. Permission denied looks like file system permissions. ls -l and touch show we can write to this directory.
As we are running on macOS, and the Dune is invoking patch, my thought goes to Apple’s patch vs GNU’s patch. Editing vendor/dune to use gpatch rather than patch allows the project to build.
$ dune build
+(cd _build/default/vendor && /usr/local/bin/gpatch --directory bitwuzla -p1) < _build/default/vendor/patch/0001-api-Add-hook-for-ocaml-z-value.patch
+File include/bitwuzla/cpp/bitwuzla.h is read-only; trying to patch anyway
+patching file include/bitwuzla/cpp/bitwuzla.h
+Running Apple’s patch directly,
$ patch -p1 < ../../../../vendor/patch/0001-api-Add-hook-for-ocaml-z-value.patch
+patching file 'include/bitwuzla/cpp/bitwuzla.h'
+Can't create '/var/folders/s_/z7_t3bvn5txfn81hk9p3ntfw0000z8/T/patchorVrfBtHVDI', output is in '/var/folders/s_/z7_t3bvn5txfn81hk9p3ntfw0000z8/T/patchorVrfBtHVDI': Permission denied
+patch: **** can't create '/var/folders/s_/z7_t3bvn5txfn81hk9p3ntfw0000z8/T/patchorVrfBtHVDI': Permission denied
+However, touch /var/folders/s_/z7_t3bvn5txfn81hk9p3ntfw0000z8/T/patchorVrfBtHVDI succeeds.
Looking back at the output from GNU patch, it reports that the file itself is read-only.
$ ls -l include/bitwuzla/cpp/bitwuzla.h
+-r--r--r-- 1 mac1000 admin 52280 May 8 15:05 include/bitwuzla/cpp/bitwuzla.h
+Let’s try to adjust the permissions:
+ +$ chmod 644 include/bitwuzla/cpp/bitwuzla.h
+$ patch -p1 < ../../../../vendor/patch/0001-api-Add-hook-for-ocaml-z-value.patch
+patching file 'include/bitwuzla/cpp/bitwuzla.h’
+And now, it succeeds. The issue is that GNU’s patch and Apple’s patch act differently when the file being patched is read-only. Apple’s patch gives a spurious error, while GNU’s patch emits a warning and makes the change anyway.
Updating the dune file to include chmod should both clear the warning and allow the use of the native patch.
(rule
+ (deps
+ (source_tree bitwuzla)
+ (:p0001
+ (file patch/0001-api-Add-hook-for-ocaml-z-value.patch))
+ (:p0002
+ (file patch/0002-binding-Fix-segfault-with-parallel-instances.patch)))
+ (target .bitwuzla_tree)
+ (action
+ (no-infer
+ (progn
+ (run chmod -R u+w bitwuzla)
+ (with-stdin-from
+ %{p0001}
+ (run patch -p1 --directory bitwuzla))
+ (with-stdin-from
+ %{p0002}
+ (run patch -p1 --directory bitwuzla))
+ (write-file %{target} "")))))
+As an essential last step, we need to tidy up on this machine. Exit the shell. Refer back to the log file for the job and run all the remaining ZFS commands. This is incredibly important on macOS and essential to keep the jobs database in sync with the layers.
diff --git a/data/planet/tunbury/deepseek-r1-on-a-raspberry-pi.md b/data/planet/tunbury/deepseek-r1-on-a-raspberry-pi.md new file mode 100644 index 0000000000..64a6d3d1bf --- /dev/null +++ b/data/planet/tunbury/deepseek-r1-on-a-raspberry-pi.md @@ -0,0 +1,127 @@ +--- +title: Deepseek R1 on a Raspberry Pi +description: "I\u2019ve heard a lot about Deepseek and wanted to try it for myself." +url: https://www.tunbury.org/2025/03/12/deekseek-r1-on-raspberry-pi/ +date: 2025-03-12T20:15:00-00:00 +preview_image: https://www.tunbury.org/images/pi5.jpg +authors: +- Mark Elvers +source: +ignore: +--- + +I’ve heard a lot about Deepseek and wanted to try it for myself.
+ +Using a Raspberry Pi 5 with 8GB of RAM and an NVMe, I installed Ollama:
+ +curl -fsSL https://ollama.com/install.sh | sh
+Next, I downloaded the smallest Deepseek model
+ +ollama pull deepseek-r1:1.5b
+And ran it using:
+ +ollama run deepseek-r1:1.5b
+Things went downhill after that, as I couldn’t get a complete response. +The Pi would lock up with no output on the display:
+ +$ ollama run deepseek-r1:1.5b
+>>> what is the capital of england?
+<think>
+Okay, so I need to figure out whatclient_loop: send disconnect: Broken pipe
+Switching from the Apple iPad charger I was using to an official Pi +power supply resolved this problem.
+ +$ ollama run deepseek-r1:1.5b
+>>> what is the capital of england?
+<think>
+
+</think>
+
+The capital of England is London.
+
+>>>
+However, it’s not so good with a more niche question:
+ +$ ollama run deepseek-r1:1.5b
+>>> what is OCaml?
+<think>
+
+</think>
+
+OCaml is a high-level, multi-paradigm programming language that combines
+features from several other languages and frameworks. It was created by
+Martin Abadi, the founding father of Google, and has gained popularity
+in both academic research and industrial applications.
+
+...
+How about an easier question?
+ +$ ollama run deepseek-r1:1.5b
+>>> Who invented the C programming language?
+<think>
+Okay, so I need to figure out who invented the C programming
+language. Hmm, I remember that C is a high-level programming language
+used for system programming and embedded systems because it's efficient
+and portable. But I'm not sure about its origin.
+
+First, I'll think about other languages that are often mentioned
+before C. For example, C was developed by Dennis Ritchie in the late
+1960s. He wrote the header files, which made the language more readable
+and maintainable. He probably worked for Bell Labs since that's where
+Dennis is associated.
+
+I wonder if there were any other contributors. Maybe someone else helped
+develop parts of the language before Ritchie did everything. But I don't
+recall any major alternatives to C being widely used before it became
+dominant in certain areas like operating systems and embedded devices.
+
+So, putting it all together, Dennis Ritchie is credited with inventing
+C. He created a lot of header files that made the language structured
+and easier to use, which helped it become very popular for its efficiency
+and portability.
+</think>
+
+The C programming language was invented by Dennis Ritchie, who worked at
+Bell Labs in the late 1960s. His work led to the creation of significant
+header files that enhanced readability and maintainability. Ritchie's
+contributions made C a widely used language in areas like operating
+systems and embedded systems.
+
+**Answer:**
+
+Dennis Ritchie invented the C programming language. He worked at Bell Labs
+(now part of Tufts University) in the late 1960s, contributing to its
+development by creating essential header files that made the language
+more readable and maintainable. His work led to a widely popularized
+language used for system programming and embedded systems.
+
+**Step-by-Step Explanation:**
+
+1. **Identify the Problem:** The question is about the origin of C,
+an early high-level programming language.
+2. **Recall Other Languages Before C:** Mention languages like Pascal,
+Ada, and Modula-2 that were around before C.
+3. **Consider Developers Known for C:** Dennis Ritchie stands out as a
+key figure in C's development.
+4. **Details about Ritchie’s Contributions:** He worked at Bell Labs
+(now Tufts University) in the 1960s, creating header files to improve
+C's structure and readability.
+5. **Link to C Language Development:** This led to C becoming prominent
+for system programming due to its efficiency and portability.
+
+**Conclusion:**
+Dennis Ritchie is credited with inventing C by developing header files
+that enhanced the language’s usability and structure.
+Over the last few months, I have written several posts on the package installation graphs specifically, Topological Sort of Packages, Installation order for opam packages and Transitive Reduction of Package Graph. In this post, I’d like to cover a alternative ordering solution.
+ +Considering the graph above, first presented in the Topological Sort of Packages, which produces the installation order below.
+ +-
+
- base-threads.base +
- base-unix.base +
- ocaml-variants +
- ocaml-config +
- ocaml +
- dune +
The code presented processed nodes when all their dependencies are satisfied (i.e., when their in-degree becomes 0). This typically means we process “leaf” nodes (nodes with no dependencies) first and then work our way up. However, it may make sense to process the leaf packages only when required rather than as soon as they can be processed. The easiest way to achieve this is to reverse the edges in the DAG, perform the topological sort, and then install the pages in reverse order.
+ +let reverse_dag (dag : PackageSet.t PackageMap.t) : PackageSet.t PackageMap.t =
+ let initial_reversed = PackageMap.fold (fun package _ acc ->
+ PackageMap.add package PackageSet.empty acc
+ ) dag PackageMap.empty in
+ PackageMap.fold (fun package dependencies reversed_dag ->
+ PackageSet.fold (fun dependency acc ->
+ let current_dependents = PackageMap.find dependency acc in
+ PackageMap.add dependency (PackageSet.add package current_dependents) acc
+ ) dependencies reversed_dag
+ ) dag initial_reversed
+With such a function, we can write this:
+ +reverse_dag dune |> topological_sort |> List.rev
+-
+
- ocaml-variants +
- ocaml-config +
- ocaml +
- base-unix.base +
- base-threads.base +
- dune +
Now, we don’t install base-unix and base-threads until they are actually required for the installation of dune.
diff --git a/data/planet/tunbury/docker-container-for-oxcaml.md b/data/planet/tunbury/docker-container-for-oxcaml.md new file mode 100644 index 0000000000..625f16615a --- /dev/null +++ b/data/planet/tunbury/docker-container-for-oxcaml.md @@ -0,0 +1,31 @@ +--- +title: Docker Container for OxCaml +description: Jon asked me to make a Docker image that contains OxCaml ready to run + without the need to build it from scratch. +url: https://www.tunbury.org/2025/07/18/docker-oxcaml/ +date: 2025-07-18T18:00:00-00:00 +preview_image: https://www.tunbury.org/images/oxcaml.png +authors: +- Mark Elvers +source: +ignore: +--- + +Jon asked me to make a Docker image that contains OxCaml ready to run without the need to build it from scratch.
+ +I have written a simple OCurrent pipeline to periodically poll oxcaml/opam-repository. If the SHA has changed, it builds a Docker image and pushes it to current/opam-staging:oxcaml.
+ +The resulting image can be run like this:
+ +$ docker run --rm -it ocurrent/opam-staging:oxcaml
+ubuntu@146eab4efc18:/$ ocaml
+OCaml version 5.2.0+ox
+Enter
+#help;; for help.
+
+#
+The exact content of the image may change depending upon requirements, and we should also pick a better place to put it rather than ocurrent/opam-staging!
+ +The pipeline code is available here mtelvers/docker-oxcaml and the service is deployed at oxcaml.image.ci.dev.
diff --git a/data/planet/tunbury/equinix-moves.md b/data/planet/tunbury/equinix-moves.md new file mode 100644 index 0000000000..2ca7b6d8e5 --- /dev/null +++ b/data/planet/tunbury/equinix-moves.md @@ -0,0 +1,27 @@ +--- +title: Equinix Moves +description: The moves of registry.ci.dev, opam-repo-ci, and get.dune.build have followed + the template of OCaml-CI. Notable differences have been that I have hosted get.dune.build + in a VM, as the services required very little disk space or CPU/RAM. For opam-repo-ci, + the rsync was pretty slow, so I tried running multiple instances using GNU parallel + with marginal gains. +url: https://www.tunbury.org/2025/04/29/equinix-moves/ +date: 2025-04-29T00:00:00-00:00 +preview_image: https://www.tunbury.org/images/equinix.png +authors: +- Mark Elvers +source: +ignore: +--- + +The moves of registry.ci.dev, opam-repo-ci, and get.dune.build have followed the template of OCaml-CI. Notable differences have been that I have hosted get.dune.build in a VM, as the services required very little disk space or CPU/RAM. For opam-repo-ci, the rsync was pretty slow, so I tried running multiple instances using GNU parallel with marginal gains.
cd /var/lib/docker/volumes2/opam-repo-ci_data/_data/var/job
+ls -d * | parallel -j 5 rsync -azh c2-4.equinix.ci.dev:/var/lib/docker/volumes/opam-repo-ci_data/_data/var/job/{}/ {}/
+The Ansible configuration script for OCaml-CI is misnamed as it configures the machine and deploys infrastructure: Caddy, Grafana, Prometheus and Docker secrets, but not the Docker stack. The Docker stack for OCaml-CI is deployed by make deploy-stack from ocurrent/ocaml-ci. Conversely, opam-repo-ci is deployed from the Ansible playbook, but there is a Makefile and an outdated stack.yml in ocurrent/opam-repo-ci.
As part of the migration away from Equinix, these services have been merged into a single large machine chives.caelum.ci.dev. With this change, I have moved the Docker stack configuration for opam-repo-ci back to the repository PR#428 and merged and renamed the machine configuration PR#44.
We want to thank Equinix for supporting OCaml over the years.
diff --git a/data/planet/tunbury/fix-opam2web-deployment.md b/data/planet/tunbury/fix-opam2web-deployment.md new file mode 100644 index 0000000000..f9e911ff28 --- /dev/null +++ b/data/planet/tunbury/fix-opam2web-deployment.md @@ -0,0 +1,130 @@ +--- +title: Fix opam2web deployment +description: We maintain a mirror (archive) of all opam packages. To take advantage + of this, add the archive mirror to opam by setting the global option. +url: https://www.tunbury.org/2025/05/28/opam2web/ +date: 2025-05-28T00:00:00-00:00 +preview_image: https://www.tunbury.org/images/opam.png +authors: +- Mark Elvers +source: +ignore: +--- + +We maintain a mirror (archive) of all opam packages. To take advantage of this, add the archive mirror to opam by setting the global option.
+ +opam option --global 'archive-mirrors+="https://opam.ocaml.org/cache"'
+How is the mirror generated and maintained?
+ +opam has a command that generates the mirror, which defaults to reading packages from the current directory.
opam admin cache --link=archives ./cache
+The base image builder pulls ocaml/opam:archive, runs opam admin cache to update the cache, and then pushes it back ocaml/opam:archive.
opam2web, which publishes opam.ocaml.org, pulls ocaml/opam:archive and then runs opam admin cache to populate any new items in the cache and then makes the cache available at https://opam.ocaml.org/cache.
Until today, the step indicated by the dotted line was missing. Kate had pointed this out as long ago as 2023 with issue #249 and PR #248, but, for whatever reason, this was never actioned.
+ +With the current unavailability of camlcity.org, this has become a problem. On Monday, I patched opam2web’s Dockerfile to include access to the mirror/cache, which allowed opam2web to build. However, subsequent builds failed because the updated opam.ocaml.org used the latest version of ocaml/opam:archive. This was built on Sunday when camlcity.org was down; therefore, the source for ocamlfind had been dropped from the mirror.
How to do we get out of this problem?
+ +Updating the base image builder does not fix the problem, as camlcity.org is still down and the current ocaml/opam:archive does not contain the missing packages. We only tag the latest version on Dockerhub, but looking through the base image builder logs allowed me to find the SHA256 for last week’s build. ocaml/opam:archive@sha256:a0e2cd50e1185fd9a17a193f52d17981a6f9ccf0b56285cbc07f396d5e3f7882
Taking PR #248, and pointing it to the older image, I used the base image builder locally to push an updated ocaml/opam:archive. This is ocaml/opam:archive@sha256:fb7b62ee305b0b9fff82748803e57a655ca92130ab8624476cd7af428101a643.
- from ~alias:"opam-archive" "ocaml/opam:archive" @@
++ from ~alias:"opam-archive" "ocaml/opam:archive@sha256:a0e2cd50e1185fd9a17a193f52d17981a6f9ccf0b56285cbc07f396d5e3f7882" @@
+Now I need to update opam.ocaml.org, but opam2web doesn’t build due to the missing ocamlfind. Checking the opam file showed two source files are needed. One is on GitHub so that’ll be ok.
...
+url {
+ src: "http://download.camlcity.org/download/findlib-1.9.6.tar.gz"
+ checksum: [
+ "md5=96c6ee50a32cca9ca277321262dbec57"
+ "sha512=cfaf1872d6ccda548f07d32cc6b90c3aafe136d2aa6539e03143702171ee0199add55269bba894c77115535dc46a5835901a5d7c75768999e72db503bfd83027"
+ ]
+}
+available: os != "win32"
+extra-source "0001-Harden-test-for-OCaml-5.patch" {
+ src:
+ "https://raw.githubusercontent.com/ocaml/opam-source-archives/main/patches/ocamlfind/0001-Harden-test-for-OCaml-5.patch"
+ checksum: [
+ "sha256=6fcca5f2f7abf8d6304da6c385348584013ffb8602722a87fb0bacbab5867fe8"
+ "md5=3cddbf72164c29d4e50e077a92a37c6c"
+ ]
+}
+Luck was on my side, as find ~/.opam/download-cache/ -name 96c6ee50a32cca9ca277321262dbec57 showed that I had the source in my local opam download cache. I checked out opam2web, copied in the file 96c6ee50a32cca9ca277321262dbec57 and patched the Dockerfile to inject it into the cache:
diff --git i/Dockerfile w/Dockerfile
+index eaf0567..84c9db8 100644
+--- i/Dockerfile
++++ w/Dockerfile
+@@ -34,6 +34,7 @@ RUN sudo mkdir -p /usr/local/bin \
+ && sudo chmod a+x /usr/local/bin/man2html
+ RUN sudo mv /usr/bin/opam-2.3 /usr/bin/opam && opam update
+ RUN opam option --global 'archive-mirrors+="https://opam.ocaml.org/cache"'
++COPY 96c6ee50a32cca9ca277321262dbec57 /home/opam/.opam/download-cache/md5/96/96c6ee50a32cca9ca277321262dbec57
+ RUN opam install odoc
+ RUN git clone https://github.com/ocaml/opam --single-branch --depth 1 --branch master /home/opam/opam
+ WORKDIR /home/opam/opam
+The final step is to build and deploy an updated opam2web incorporating the updated mirror cache. In conjunction with the updated base image builder, this will be self-sustaining. I wrapped the necessary steps into a Makefile.
OPAM_REPO_GIT_SHA := $(shell git -C ~/opam-repository fetch upstream && git -C ~/opam-repository rev-parse upstream/master)
+BLOG_GIT_SHA := bdef1bbf939db6797dcd51faef2ea9ac1826f4a5
+OPAM_GIT_SHA := 46234090daf4f9c5f446af56a50f78809c04a20a
+
+all: opam2web
+ cd opam2web && docker --context registry.ci.dev build --pull \
+ --build-arg OPAM_REPO_GIT_SHA=$(OPAM_REPO_GIT_SHA) \
+ --build-arg BLOG_GIT_SHA=$(BLOG_GIT_SHA) \
+ --build-arg OPAM_GIT_SHA=$(OPAM_GIT_SHA) \
+ -f Dockerfile --iidfile ../docker-iid -- .
+ @SHA256=$$(cat docker-iid)
+ docker --context registry.ci.dev tag $$SHA256 registry.ci.dev/opam.ocaml.org:live
+ docker --context registry.ci.dev login -u $(USERNAME) -p $(PASSWORD) registry.ci.dev
+ docker --context registry.ci.dev push registry.ci.dev/opam.ocaml.org:live
+ docker --context opam-4.ocaml.org pull registry.ci.dev/opam.ocaml.org:live
+ docker --context opam-4.ocaml.org service update infra_opam_live --image $$SHA256
+ docker --context opam-5.ocaml.org pull registry.ci.dev/opam.ocaml.org:live
+ docker --context opam-5.ocaml.org service update infra_opam_live --image $$SHA256
+
+opam2web:
+ git clone --recursive "https://github.com/ocaml-opam/opam2web.git" -b "live"
+Check that ocamlfind is included in the new cache
wget https://opam-4.ocaml.org/cache/md5/96/96c6ee50a32cca9ca277321262dbec57
+wget https://opam-5.ocaml.org/cache/md5/96/96c6ee50a32cca9ca277321262dbec57
+
+CI workers spring and summer run FreeBSD and need to be updated.
Check the current version of FreeBSD which we have with uname -r.
FreeBSD summer 14.1-RELEASE-p5 FreeBSD 14.1-RELEASE-p5 GENERIC amd64
+Run freebsd-update fetch to download the latest versions of the system components, particularly the freebsd-update utility. It even reported that it really is time to upgrade!
# freebsd-update fetch
+...
+WARNING: FreeBSD 14.1-RELEASE-p5 is approaching its End-of-Life date.
+It is strongly recommended that you upgrade to a newer
+release within the next 5 days.
+Install these updates.
+ +freebsd-update install
+Now use freebsd-update to fetch the 14.2-RELEASE and install it.
# freebsd-update upgrade -r 14.2-RELEASE
+...
+# freebsd-update install
+src component not installed, skipped
+Installing updates...
+Kernel updates have been installed. Please reboot and run
+'freebsd-update [options] install' again to finish installing updates.
+Reboot the system using reboot and then finish installing updates.
# freebsd-update install
+src component not installed, skipped
+Installing updates...
+Restarting sshd after upgrade
+Performing sanity check on sshd configuration.
+Stopping sshd.
+Waiting for PIDS: 707.
+Performing sanity check on sshd configuration.
+Starting sshd.
+Scanning /usr/share/certs/untrusted for certificates...
+Scanning /usr/share/certs/trusted for certificates...
+Scanning /usr/local/share/certs for certificates...
+ done.
+Now use pkg to upgrade any applications.
# pkg upgrade
+Updating FreeBSD repository catalogue...
+Fetching data.pkg: 100% 7 MiB 7.5MB/s 00:01
+Processing entries: 100%
+FreeBSD repository update completed. 35885 packages processed.
+All repositories are up to date.
+Checking for upgrades (28 candidates): 100%
+Processing candidates (28 candidates): 100%
+The following 28 package(s) will be affected (of 0 checked):
+
+Installed packages to be UPGRADED:
+ curl: 8.10.1 -> 8.11.1_1
+...
+ xxd: 9.1.0764 -> 9.1.1199
+
+Number of packages to be upgraded: 28
+
+The process will require 3 MiB more space.
+77 MiB to be downloaded.
+
+Proceed with this action? [y/N]: y
+Finally, reboot the system and check uname -a.
# uname -a
+FreeBSD spring 14.2-RELEASE-p1 FreeBSD 14.2-RELEASE-p1 GENERIC amd64
+To update the the FreeBSD base images used by the CI services, I applied PR#13 to ocurrent/freebsd-infra.
+ +This was followed up by PR#1007 on ocurrent/ocaml-ci and PR#427 to ocurrent/opam-repo-ci.
diff --git a/data/planet/tunbury/improve-the-deployment-time-for-opam2web.md b/data/planet/tunbury/improve-the-deployment-time-for-opam2web.md new file mode 100644 index 0000000000..c1c97d130d --- /dev/null +++ b/data/planet/tunbury/improve-the-deployment-time-for-opam2web.md @@ -0,0 +1,135 @@ +--- +title: Improve the deployment time for opam2web +description: The opam2web image for opam.ocaml.org is huge weighing in at more than + 25 GB. The bulk of this data is opam archives, which are updated and copied into + a stock caddy image. +url: https://www.tunbury.org/2025/06/24/opam2web/ +date: 2025-06-24T00:00:00-00:00 +preview_image: https://www.tunbury.org/images/opam.png +authors: +- Mark Elvers +source: +ignore: +--- + +The opam2web image for opam.ocaml.org is huge weighing in at more than 25 GB. The bulk of this data is opam archives, which are updated and copied into a stock caddy image.
+ +There are two archives, ocaml/opam.ocaml.org-legacy, which hasn’t changed for 5 years and holds the cache for opam 1.x and ocaml/opam:archive, which is updated weekly.
The current Dockerfile copies these files into a new layer each time opam2web builds.
FROM --platform=linux/amd64 ocaml/opam:archive as opam-archive
+FROM ocaml/opam.ocaml.org-legacy as opam-legacy
+FROM alpine:3.20 as opam2web
+...
+COPY --from=opam-legacy . /www
+...
+RUN --mount=type=bind,target=/cache,from=opam-archive rsync -aH /cache/cache/ /www/cache/
+...
+And later, the entire /www structure is copied into a caddy:2.8.4 image.
FROM caddy:2.8.4
+WORKDIR /srv
+COPY --from=opam2web /www /usr/share/caddy
+COPY Caddyfile /etc/caddy/Caddyfile
+ENTRYPOINT ["caddy", "run", "--config", "/etc/caddy/Caddyfile", "--adapter", "caddyfile"]
+This method is considered “best practice” when creating Docker images, but in this case, it produces a very large image, which takes a long time to deploy.
+ +For Docker to use an existing layer, we need the final FROM ... to be the layer we want to use as the base. In the above snippet, the caddy:2.8.4 layer will be the base layer and will be reused.
The archive, ocaml/opam:archive, is created by this Dockerfile, which ultimately uses alpine:latest.
FROM ocaml/opam:archive AS opam-archive
+FROM ocurrent/opam-staging@sha256:f921cd51dda91f61a52a2c26a8a188f8618a2838e521d3e4afa3ca1da637903e AS archive
+WORKDIR /home/opam/opam-repository
+RUN --mount=type=bind,target=/cache,from=opam-archive rsync -aH /cache/cache/ /home/opam/opam-repository/cache/
+RUN opam admin cache --link=/home/opam/opam-repository/cache
+
+FROM alpine:latest
+COPY --chown=0:0 --from=archive [ "/home/opam/opam-repository/cache", "/cache" ]
+In our opam2web build, we could use FROM ocaml/opam:archive and then apk add caddy, which would reuse the entire 15GB layer and add the few megabytes for caddy.
ocaml/opam.ocaml.org-legacy is another 8GB. This legacy data could be integrated by adding it to ocaml/opam:archive in a different directory to ensure compatibility with anyone else using this image. This is PR#324
let install_package_archive opam_image =
+ let open Dockerfile in
++ from ~alias:"opam-legacy" "ocaml/opam.ocaml.org-legacy" @@
+ from ~alias:"opam-archive" "ocaml/opam:archive" @@
+ from ~alias:"archive" opam_image @@
+ workdir "/home/opam/opam-repository" @@
+ run ~mounts:[mount_bind ~target:"/cache" ~from:"opam-archive" ()] "rsync -aH /cache/cache/ /home/opam/opam-repository/cache/" @@
+ run "opam admin cache --link=/home/opam/opam-repository/cache" @@
+ from "alpine:latest" @@
++ copy ~chown:"0:0" ~from:"opam-legacy" ~src:["/"] ~dst:"/legacy" () @@
+ copy ~chown:"0:0" ~from:"archive" ~src:["/home/opam/opam-repository/cache"] ~dst:"/cache" ()
+Finally, we need to update opam2web to use ocaml/opam:archive as the base layer rather than caddy:2.8.4, resulting in the final part of the Dockerfile looking like this.
FROM ocaml/opam:archive
+RUN apk add --update git curl rsync libstdc++ rdfind caddy
+COPY --from=build-opam2web /opt/opam2web /usr/local
+COPY --from=build-opam-doc /usr/bin/opam-dev /usr/local/bin/opam
+COPY --from=build-opam-doc /opt/opam/doc /usr/local/share/opam2web/content/doc
+COPY ext/key/opam-dev-team.pgp /www/opam-dev-pubkey.pgp
+ADD bin/opam-web.sh /usr/local/bin
+ARG DOMAIN=opam.ocaml.org
+ARG OPAM_REPO_GIT_SHA=master
+ARG BLOG_GIT_SHA=master
+RUN echo ${OPAM_REPO_GIT_SHA} >> /www/opam_git_sha
+RUN echo ${BLOG_GIT_SHA} >> /www/blog_git_sha
+RUN /usr/local/bin/opam-web.sh ${DOMAIN} ${OPAM_REPO_GIT_SHA} ${BLOG_GIT_SHA}
+WORKDIR /srv
+COPY Caddyfile /etc/caddy/Caddyfile
+ENTRYPOINT ["caddy", "run", "--config", "/etc/caddy/Caddyfile", "--adapter", "caddyfile"]
+I acknowledge that this final image now contains some extra unneeded packages such as git, curl, etc, but this seems a minor inconvenience.
The Caddyfile can be adjusted to make everything still appear to be in the same place:
:80 {
+ redir /install.sh https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh
+ redir /install.ps1 https://raw.githubusercontent.com/ocaml/opam/master/shell/install.ps1
+
+ @version_paths path /1.1/* /1.2.0/* /1.2.2/*
+ handle @version_paths {
+ root * /legacy
+ file_server
+ }
+
+ handle /cache/* {
+ root * /
+ file_server
+ }
+
+ handle {
+ root * /www
+ file_server
+ }
+}
+In this configuration, the Docker push is only 650MB rather than 25GB.
+ +The changes to opam2web are in PR#245
+ +Test with some external URLs:
+ +-
+
- https://staging.opam.ocaml.org/index.tar.gz +
- https://staging.opam.ocaml.org/archives/0install.2.18/0install-2.18.tbz +
- https://staging.opam.ocaml.org/cache/0install.2.18/0install-2.18.tbz +
- https://staging.opam.ocaml.org/1.2.2/archives/0install.2.12.3+opam.tar.gz +
- https://staging.opam.ocaml.org/1.2.0/archives/0install.2.12.1+opam.tar.gz +
- https://staging.opam.ocaml.org/1.1/archives/0install.2.10+opam.tar.gz +
- https://staging.opam.ocaml.org/opam_git_sha +
- https://staging.opam.ocaml.org/blog_git_sha +
- https://staging.opam.ocaml.org/opam-dev-pubkey.pgp +
Previously, I discussed the installation order for a simple directed acyclic graph without any cycles. However, opam packages include post dependencies. Rather than package A depending upon B where B would be installed first, post dependencies require X to be installed after Y. The post dependencies only occur in a small number of core OCaml packages. They are quite often empty and exist to direct the solver. Up until now, I had been using a base layer with an opam switch containing the base compiler and, therefore, did not need to deal with any post dependencies.
Here is the graph of 0install with post dependencies coloured in red.
+ +Removing the post dependencies gives an unsatisfying graph with orphaned dependencies. 0install without post. Note base-nnp.base and base-effects.base. However, this graph can be used to produce a linear installation order. The orphaned packages can be removed with a recursive search.
When opam wants to decide the installation order, it uses OCamlgraph’s topological sort capability.
+ +++ +This functor provides functions which allow iterating over a graph in topological order. Cycles in graphs are allowed. Specification is the following: If vertex [x] is visited before vertex [y] then either there is a path from [x] to [y], or there is no path from [y] to [x]. In the particular case of a DAG, this simplifies to: if there is an edge from [x] to [y], then [x] is visited before [y].
+
The description of fold is particularly interesting as the order for cycles is unspecified.
++ +[fold action g seed] allows iterating over the graph [g] in topological order. [action node accu] is called repeatedly, where [node] is the node being visited, and [accu] is the result of the [action]’s previous invocation, if any, and [seed] otherwise. If [g] contains cycles, the order is unspecified inside the cycles and every node in the cycles will be presented exactly once
+
In my testing, the installation order matches the order used by opam within the variation allowed above.
+ +Layers can be built up using the intersection of packages installed so far and the required dependencies.
diff --git a/data/planet/tunbury/irmin-database.md b/data/planet/tunbury/irmin-database.md new file mode 100644 index 0000000000..8c3e055a5d --- /dev/null +++ b/data/planet/tunbury/irmin-database.md @@ -0,0 +1,42 @@ +--- +title: Irmin Database +description: "After Thomas\u2019 talk today I wanted to try Irmin for myself." +url: https://www.tunbury.org/2025/03/17/irmin/ +date: 2025-03-17T00:00:00-00:00 +preview_image: https://www.tunbury.org/images/irmin.png +authors: +- Mark Elvers +source: +ignore: +--- + +After Thomas’ talk today I wanted to try Irmin for myself.
+ +In a new switch I installed Irmin via opam opam install irmin-git and then built the example code
open Lwt.Syntax
+module Git_store = Irmin_git_unix.FS.KV (Irmin.Contents.String)
+module Git_info = Irmin_unix.Info (Git_store.Info)
+
+let git_config = Irmin_git.config ~bare:true "./db"
+let info message = Git_info.v ~author:"Example" "%s" message
+
+let main_branch config =
+ let* repo = Git_store.Repo.v config in
+ Git_store.main repo
+
+let main =
+ let* t = main_branch git_config in
+ (* Set a/b/c to "Hello, Irmin!" *)
+ let* () =
+ Git_store.set_exn t [ "a"; "b"; "c" ] "Hello, Irmin!"
+ ~info:(info "my first commit")
+ in
+ (* Get a/b/c *)
+ let+ s = Git_store.get t [ "a"; "b"; "c" ] in
+ assert (s = "Hello, Irmin!")
+
+let () = Lwt_main.run main
+I’m pretty excited about the possibilities.
diff --git a/data/planet/tunbury/label-maker-in-jsofocaml-using-claude.md b/data/planet/tunbury/label-maker-in-jsofocaml-using-claude.md new file mode 100644 index 0000000000..6e5e064203 --- /dev/null +++ b/data/planet/tunbury/label-maker-in-jsofocaml-using-claude.md @@ -0,0 +1,60 @@ +--- +title: Label Maker in js_of_ocaml using Claude +description: "I\u2019ve taken a few days off, and while I\u2019ve been travelling, + I\u2019ve been working on a personal project with Claude. I\u2019ve used Claude + Code for the first time, which is a much more powerful experience than using claude.ai + as Claude can apply changes to the code and use your build tools directly to quickly + iterate on a problem. In another first, I used js_of_ocaml, which has been awesome." +url: https://www.tunbury.org/2025/08/22/label-maker/ +date: 2025-08-22T00:00:00-00:00 +preview_image: https://www.tunbury.org/images/ocaml-logo.png +authors: +- Mark Elvers +source: +ignore: +--- + +I’ve taken a few days off, and while I’ve been travelling, I’ve been working on a personal project with Claude. I’ve used Claude Code for the first time, which is a much more powerful experience than using claude.ai as Claude can apply changes to the code and use your build tools directly to quickly iterate on a problem. In another first, I used js_of_ocaml, which has been awesome.
The project isn’t anything special; it’s a website that creates sheets of Avery labels. It is needed for a niche educational environment where the only devices available are iPads, which are administratively locked down, so no custom applications or fonts can be loaded. You enter what you want on the label, and it initiates the download of the resulting PDF.
+ +The original implementation, written in OCaml (of course), uses a cohttp web server, which generates a reStructuredText file which is processed via rst2pdf with custom page templates for the different label layouts. The disadvantage of this approach is that it requires a server to host it. I have wrapped the application into a Docker container, so it isn’t intrusive, but it would be easier if it could be hosted as a static file on GitHub Pages.
+ +On OCaml.org, I found camlpdf, otfm and vg, which when combined with js_of_ocaml, should give me a complete tool in the browser. The virtual file system embeds the TTF font into the JavaScript code!
I set Claude to work, which didn’t take long, but the custom font embedding proved problematic. I gave Claude an example PDF from the original implementation, and after some debugging, we had a working project.
+ +Let’s look at the code! I should add that the labels can optionally have a box drawn on them, which the student uses to provide feedback on how they got on with the objective. Claude produced three functions for rendering text: one for a single line, one for multiline text with a checkbox, and one for multiline text without a checkbox. I pointed out that these three functions were similar and could be combined. Claude agreed and created a merged function with the original three functions calling the new merged function. It took another prompt to update the calling locations to call the new merged function rather than having the stub functions.
+ +While Claude had generated code that compiles in a functional language, the code tends to look imperative; for example, there were several instances like this:
+ +let t = ref 0 in
+let () = List.iter (fun v -> t := !t + v) [1; 2; 3] in
+t
+Where we would expect to see a List.fold_left! Claude can easily fix these when you point them out.
As I mentioned earlier, Claude code can build your project and respond to dune build errors for you; however, some fixes suppress the warning rather than actually fixing the root cause. A classic example of this is:
% dune build
+File "bin/main.ml", line 4, characters 4-5:
+4 | let x = List.length lst
+ ^
+Error (warning 32 [unused-value-declaration]): unused value x.
+The proposed fix is to discard the value of x, thus let _x = List.length lst rather than realising that the entire line is unnecessary as List.length has no side effects.
I’d been using Chrome 139 for development, but thought I’d try in the native Safari on my Monterey-based based MacPro which has Safari 17.6. This gave me this error on the JavaScript console.
+ +[Error] TypeError: undefined is not
+ an object (evaluating 'k.UNSIGNED_MAX.udivmod')
+ db (label_maker.bc.js:1758)
+ (anonymous function) (label_maker.bc.js:1930)
+ Global Code (label_maker.bc.js:2727:180993)
+I found that since js_of_ocaml 6.0.1 the minimum browser version is Safari 18.2, so I switched to js_of_ocaml 5.9.1 and that worked fine.
The resulting project can be found at mtelvers/label-maker-js and published at mtelvers.github.io/label-maker-js.
diff --git a/data/planet/tunbury/moving-ocaml-ci.md b/data/planet/tunbury/moving-ocaml-ci.md new file mode 100644 index 0000000000..0e5fb952d0 --- /dev/null +++ b/data/planet/tunbury/moving-ocaml-ci.md @@ -0,0 +1,45 @@ +--- +title: Moving OCaml-CI +description: As noted on Thursday, the various OCaml services will need to be moved + away from Equinix. Below are my notes on moving OCaml-CI. +url: https://www.tunbury.org/2025/04/27/ocaml-ci/ +date: 2025-04-27T00:00:00-00:00 +preview_image: https://www.tunbury.org/images/ocaml-logo.png +authors: +- Mark Elvers +source: +ignore: +--- + +As noted on Thursday, the various OCaml services will need to be moved away from Equinix. Below are my notes on moving OCaml-CI.
+ +Generate an SSH key on the new server chives using ssh-keygen -t ed25519. Copy the public key to c2-3.equinix.ci.dev and save it under ~/.ssh/authorized_keys.
Use rsync to mirror the Docker volumes. -z did improve performance as there appears to be a rate limiter somewhere in the path.
rsync -azvh --progress c2-3.equinix.ci.dev:/var/lib/docker/volumes/ /var/lib/docker/volumes/
+After completing the copy, I waited for a quiet moment, and then scaled all of the Docker services to 0. I prefer to scale the services rather than remove them, as the recovery is much easier.
+ +docker service scale infra_grafana=0
+docker service scale infra_prometheus=0
+docker service scale ocaml-ci_ci=0
+docker service scale ocaml-ci_gitlab=0
+docker service scale ocaml-ci_web=0
+For the final copy, I used --checksum and also added --delete, as the Prometheus database creates segment files that are periodically merged into the main database.
rsync -azvh --checksum --delete --progress c2-3.equinix.ci.dev:/var/lib/docker/volumes/ /var/lib/docker/volumes/
+The machine configuration is held in an Ansible Playbook, which includes the Docker stack for Grafana and Prometheus. It can be easily applied to the new machine:
+ +ansible-playbook -e @secrets/ocaml.ci.dev.yml --vault-password-file secrets/vault-password ocaml.ci.dev.yml
+OCaml-CI’s Docker stack is held on GitHub ocurrent/ocaml-ci and can be deployed with:
+ +make deploy-stack
+opam 2.4.0 was released on 18th July followed by opam 2.4.1 a few days later. This update needs to be propagated through the CI infrastructure. The first step is to update the base images for each OS.
+ +Linux
+ +ocurrent/docker-base-images
+ +The Linux base images are created using the Docker base image builder, which uses ocurrent/ocaml-dockerfile to know which versions of opam are available. Kate submitted PR#235 with the necessary changes to ocurrent/ocaml-dockerfile. This was released as v8.2.9 under PR#28251.
+ +With v8.2.9 released, PR#327 can be opened to update the pipeline to build images which include opam 2.4. Rebuilding the base images takes a good deal of time, particularly as it’s marked as a low-priority task on the cluster.
+ +macOS
+ +ocurrent/macos-infra
+ +Including opam 2.4 in the macOS required PR#56, which adds 2.4.1 to the list of opam packages to download. There are Ansible playbooks that build the macOS base images and recursively remove the old images and their (ZFS) clones. They take about half an hour per machine. I run the Intel and Apple Silicon updates in parallel, but process each pool one at a time.
The Ansible command is:
+ +ansible-playbook update-ocluster.yml
+FreeBSD (rosemary.caelum.ci.dev)
+ +ocurrent/freebsd-infra
+ +The FreeBSD update parallels the macOS update, requiring that 2.4.1 be added to the loop of available versions. PR#15.
The Ansible playbook for updating the machine is named update.yml. However, we have been suffering from some reliability issues with the FreeBSD worker, see issue#449, so I took the opportunity to rebuild the worker from scratch.
The OS reinstallation is documented in this post, and it’s definitely worth reading the README.md in the repo for the post-installation steps.
+ +Windows (thyme.caelum.ci.dev)
+ +ocurrent/obuilder
+ +The Windows base images are built using a Makefile which runs unattended builds of Windows using QEMU virtual machines. The Makefile required PR#198 to The command is make windows.
Once the new images have been built, stop ocluster worker and move the new base images into place.
+The next is to remove results/* as these layers will link to the old base images, and remove state/* so obuilder will create a new empty database on startup. Avoid removing cache/* as this is the download cache for opam objects.
The unattended installation can be monitored via VNC by connecting to localhost:5900.
+ +OpenBSD (oregano.caelum.ci.dev)
+ +ocurrent/obuilder
+ +The OpenBSD base images are built using the same Makefile used for Windows. There is a seperate commit in PR#198 for the changes needed for OpenBSD, which include moving from OpenBSD 7.6 to 7.7. Run make openbsd.
Once the new images have been built, stop ocluster worker and move the new base images into place.
+The next is to remove results/* as these layers will link to the old base images, and remove state/* so obuilder will create a new empty database on startup. Avoid removing cache/* as this is the download cache for opam objects.
As with Windows, the unattended installation can be monitored via VNC by connecting to localhost:5900.
+ +OCaml-CI
+ +OCaml-CI uses ocurrent/ocaml-dockerfile as a submodule, so the module needs to be updated to the released version. Edits are needed to lib/opam_version.ml to include V2_4, then the pipeline needs to be updated in service/conf.ml to use version 2.4 rather than 2.3 for all the different operating systems. Linux is rather more automated than the others
Lastly, since we now have OpenBSD 7.7, I have also updated references to OpenBSD 7.6. PR#1020.
+ +opam-repo-ci
+ +opam-repo-ci tests using the latest tagged version of opam, which is called opam-dev within the base images. It also explicitly tests against the latest release in each of the 2.x series. With 2.4 being tagged, this will automatically become the used dev version once the base images are updated, but over time, 2.4 and the latest tagged version will diverge, so PR#448 is needed to ensure we continue to test with the released version of 2.4.
In my OCaml project, I’d like to abstract away the details of running containers into specific modules based on the OS. Currently, I have working container setups for Windows and Linux, and I’ve haphazardly peppered if Sys.win32 then where I need differentiation, but this is OCaml, so let us use functors!
I started by fleshing out the bare bones in a new project. After dune init project functor, I created bin/s.ml containing the signature of the module CONTAINER.
module type CONTAINER = sig
+ val run : string -> unit
+end
+Then a trivial bin/linux.ml.
let run s = Printf.printf "Linux container '%s'\n" s
+And bin/windows.ml.
let run s = Printf.printf "Windows container '%s'\n" s
+Then in bin/main.ml, I can select the container system once and from then on use Container.foo to run the appropriate OS specific function.
let container = if Sys.win32 then (module Windows : S.CONTAINER) else (module Linux : S.CONTAINER)
+
+module Container = (val container)
+
+let () = Container.run "Hello, World!"
+You can additionally create windows.mli and linux.mli containing simply include S.CONTAINER.
Now, let’s imagine that we needed to have some specific configuration options depending upon whether we are running on Windows or Linux. For demonstration purposes, let’s use the user account. On Windows, this is a string, typically ContainerAdministrator, whereas on Linux, it’s an integer UID of value 0.
We can update the module type in bin/s.ml to include the type t, and add an init function to return a t and add t as a parameter to run.
module type CONTAINER = sig
+ type t
+
+ val init : unit -> t
+ val run : t -> string -> unit
+end
+In bin/linux.ml, we can add the type and define uid as an integer, then add the init function to return the populated structure. run now accepts t as the first parameter.
type t = {
+ uid : int;
+}
+
+let init () = { uid = 0 }
+
+let run t s = Printf.printf "Linux container user id %i says '%s'\n" t.uid s
+In a similar vein, bin/windows.ml is updated like this
type t = {
+ username : string;
+}
+
+let init () = { username = "ContainerAdministrator" }
+
+let run t s = Printf.printf "Windows container user name %s says '%s'\n" t.username s
+And finally, in bin/main.ml we run Container.init () and use the returned type as a parameter to Container.run.
let container = if Sys.win32 then (module Windows : S.CONTAINER) else (module Linux : S.CONTAINER)
+
+module Container = (val container)
+
+let c = Container.init ()
+let () = Container.run c "Hello, World!"
+Yesterday, we were talking about extending the current infrastructure database to incorporate other information to provide prompts to return machines to the pool of resources after they have completed their current role/loan, etc. There is also a wider requirement to bring these services back to Cambridge from Equinix/Scaleway, which will be the subject of a follow-up post. However, the idea of extending the database made me think that it would be amusing to overlay the machine’s positions onto Google Maps.
+ +I added positioning data in the Jekyll Collection _machines\*.md for each machine. e.g. ainia.md
---
+name: ainia
+...
+latitude: 52.2109
+longitude: 0.0917
+---
+Then Jekyll’s Liquid templating engine can create a JavaScript array for us
+ +
+ // Define machines data array from Jekyll collection
+ const machinesData = [
+ {% for machine in site.machines %}
+ {% if machine.latitude and machine.longitude %}
+ {
+ name: "{{ machine.name }}",
+ lat: {{ machine.latitude }},
+ lng: {{ machine.longitude }},
+ {% if machine.description %}
+ description: "{{ machine.description | escape }}",
+ {% endif %}
+ // Add any other properties you need
+ },
+ {% endif %}
+ {% endfor %}
+ ];
+
+This array can be converted into an array of map markers. Google have an API for clustering the markers into a count of machines. I added a random offset to each location to avoid all the markers piling up on a single spot.
+ +The interactive map can be seen at machines.html
diff --git a/data/planet/tunbury/ocaml-lt-414-fedora-42-and-gcc-15.md b/data/planet/tunbury/ocaml-lt-414-fedora-42-and-gcc-15.md new file mode 100644 index 0000000000..74d93ac5b8 --- /dev/null +++ b/data/planet/tunbury/ocaml-lt-414-fedora-42-and-gcc-15.md @@ -0,0 +1,63 @@ +--- +title: OCaml < 4.14, Fedora 42 and GCC 15 +description: Late last week, @MisterDA added Fedora 42 support to the Docker base + image builder. The new base images attempted to build over the weekend, but there + have been a few issues! +url: https://www.tunbury.org/2025/04/22/ocaml-fedora-gcc/ +date: 2025-04-22T00:00:00-00:00 +preview_image: https://www.tunbury.org/images/ocaml-logo.png +authors: +- Mark Elvers +source: +ignore: +--- + +Late last week, @MisterDA added Fedora 42 support to the Docker base image builder. The new base images attempted to build over the weekend, but there have been a few issues!
+ +The code I had previously added to force Fedora 41 to use the DNF version 5 syntax was specifically for version 41. For reference, the old syntax was yum groupinstall -y 'C Development Tools and Libraries’, and the new syntax is yum group install -y 'c-development'. Note the extra space.
let c_devtools_libs : (t, unit, string, t) format4 =
+ match d with
+ | `Fedora `V41 -> {|"c-development"|}
+ | `Fedora _ -> {|"C Development Tools and Libraries"|}
+ | _ -> {|"Development Tools”|}
+...
+let dnf_version = match d with `Fedora `V41 -> 5 | _ -> 3
+To unburden ourselves of this maintenance in future releases, I have inverted the logic so unmatched versions will use the new syntax.
+ +let (dnf_version, c_devtools_libs) : int * (t, unit, string, t) format4 =
+ match d with
+ | `Fedora
+ ( `V21 | `V22 | `V23 | `V24 | `V25 | `V26 | `V27 | `V28 | `V29
+ | `V30 | `V31 | `V32 | `V33 | `V34 | `V35 | `V36 | `V37 | `V38
+ | `V39 | `V40 ) ->
+ (3, {|"C Development Tools and Libraries"|})
+ | `Fedora _ -> (5, {|"c-development"|})
+ | _ -> (3, {|"Development Tools"|})
+Fedora 42 also removed awk, so it now needs to be specifically included as a dependency. However, this code is shared with Oracle Linux, which does not have a package called awk. Fortunately, both have a package called gawk!
The next issue is that Fedora 42 is the first of the distributions we build base images for that has moved to GCC 15, specifically GCC 15.0.1. This breaks all versions of OCaml < 4.14.
+ +The change is that the code below, which previously gave no information about the number or type of parameters. (see runtime/caml/prims.h)
typedef value (*c_primitive)();
+Now means that there are no parameters, aka:
+ +typedef value (*c_primitive)(void);
+This is caused by a change of the default compilter language version. See GCC change log
+ +++ +C23 by default: GCC 15 changes the default language version for C compilation from
+-std=gnu17to-std=gnu23. If your code relies on older versions of the C standard, you will need to either add-std=to your build flags, or port your code; see the porting notes.
Also see the porting notes, and this bug report.
+ +This is not an immediate problem as OCaml-CI and opam-repo-ci only test against OCaml 4.14.2 and 5.3.0 on Fedora. I have opened issue#320 to track this problem.
diff --git a/data/planet/tunbury/ocaml-program-specification-for-claude.md b/data/planet/tunbury/ocaml-program-specification-for-claude.md new file mode 100644 index 0000000000..84aab5d286 --- /dev/null +++ b/data/planet/tunbury/ocaml-program-specification-for-claude.md @@ -0,0 +1,79 @@ +--- +title: OCaml Program Specification for Claude +description: I have a dataset that I would like to visualise using a static website + hosted on GitHub Pages. The application that generates the dataset is still under + development, which results in frequently changing data formats. Therefore, rather + than writing a static website generator and needing to revise it continually, could + I write a specification and have Claude create a new one each time there was a change? +url: https://www.tunbury.org/2025/08/01/program-specification/ +date: 2025-08-01T00:00:00-00:00 +preview_image: https://www.tunbury.org/images/ocaml-logo.png +authors: +- Mark Elvers +source: +ignore: +--- + +I have a dataset that I would like to visualise using a static website hosted on GitHub Pages. The application that generates the dataset is still under development, which results in frequently changing data formats. Therefore, rather than writing a static website generator and needing to revise it continually, could I write a specification and have Claude create a new one each time there was a change?
+ +Potentially, I could do this cumulatively by giving Claude the original specification and code and then the new specification, but my chosen approach is to see if Claude can create the application in one pass from the specification. I’ve also chosen to do this using Claude Sonnet’s web interface; obviously, the code I will request will be in OCaml.
+ +I wrote a detailed 500-word specification that included the file formats involved, example directory tree layouts, and what I thought was a clear definition of the output file structure.
+ +The resulting code wasn’t what I wanted: Claude had inlined huge swathes of HTML and was using Printf.sprintf extensively. Each file included the stylesheet as a <style>...</style>. However, the biggest problem was that Claude had chosen to write the JSON parser from scratch, and this code had numerous issues and wouldn’t even build. I directed Claude to use yojson rather than handcraft a parser.
I intended but did not state in my specification that I wanted the code to generate HTML using tyxml. I updated my specification, requesting that the code be written using tyxml, yojson, and timedesc to handle the ISO date format. I also thought of some additional functionality around extracting data from a Git repo.
Round 2 - Possibly a step backwards as Claude struggled to find the appropriate functions in the timedesc library to parse and sort dates. There were also some issues extracting data using git. I have to take responsibility here as I gave the example command as git show --date=iso-strict ce03608b4ba656c052ef5e868cf34b9e86d02aac -C /path/to/repo, but git requires the -C /path/to/repo to precede the show command. However, the fact that my example had overwritten Claude’s knowledge was potentially interesting. Could I use this to seed facts I knew Claude would need?
Claude still wasn’t creating a separate stylesheet.css.
Round 3 - This time, I gave examples on how to use the timedesc library, i.e.
++ +To use the
+timedesclibrary, we can callTimedesc.of_iso8601to convert the Git ISO strict output to a Timedesc object and then compare it withcompare (Timedesc.to_timestamp_float_s b.date) (Timedesc.to_timestamp_float_s a.date).
Also, in addition to stating that all the styles should be shared in a common stylesheet.css, I gave a file tree of the expected output, including the stylesheet.css.
Claude now correctly used the timedesc library and tried to write a stylesheet. However, Claude had hallucinated a css and css_rule function in tyxml to do this, where none exists. Furthermore, adding the link to the stylesheet was causing problems as link had multiple definitions in scope and needed to be explicitly referenced as Tyxml.Html.link. Claude’s style was to open everything at the beginning of the file:
open Yojson.Safe
+open Yojson.Safe.Util
+open Tyxml.Html
+open Printf
+open Unix
+The compiler picked Unix.link rather than Tyxml.Html.link:
File "ci_generator.ml", line 347, characters 18-33:
+347 | link ~rel:[ `Stylesheet ] ~href:"/stylesheet.css" ();
+ ^^^^^^^^^^^^^^^
+Error: The function applied to this argument has type
+ ?follow:bool -> string -> unit
+This argument cannot be applied with label ~rel
+++ +Stylistically, please can we only
+openthings in functions where they are used:let foo () = let open Tyxml.Html in .... This will avoid global opens at the top of the file and avoid any confusion where libraries have functions with the same name, e.g.,Unix.linkandTyXml.Html.link.
Furthermore, I had two JSON files in my input, each with the field name. Claude converted these into OCaml types; however, when referencing these later as function parameters, the compiler frequently picks the wrong one. This can be fixed by adding a specific type to the function parameter let f (t:foo) = .... I’ve cheated here and renamed the field in one of the JSON files.
type foo = {
+ name : string;
+ x : string;
+}
+
+type bar = {
+ name : string;
+ y : string;
+}
+Claude chose to extract the data from the Git repo using git show --pretty=format:'%H|%ai|%s', this ignores the --date=iso-strict directive. The correct format should be %aI. I updated my guidance on the use of git show.
My specification now comes in just under 1000 words. From that single specification document, Claude produces a valid OCaml program on the first try, which builds the static site as per my design. wc -l shows me there are 662 lines of code.
It’s amusing to run it more than once to see the variations in styling!
diff --git a/data/planet/tunbury/opam-health-check-with-oxcaml.md b/data/planet/tunbury/opam-health-check-with-oxcaml.md new file mode 100644 index 0000000000..fe5cb9183a --- /dev/null +++ b/data/planet/tunbury/opam-health-check-with-oxcaml.md @@ -0,0 +1,61 @@ +--- +title: Opam Health Check with OxCaml +description: "Arthur mentioned that it would be great to know which packages build + successfully with OxCaml and which don\u2019t." +url: https://www.tunbury.org/2025/05/14/opam-health-check-oxcaml/ +date: 2025-05-14T06:00:00-00:00 +preview_image: https://www.tunbury.org/images/ocaml-logo.png +authors: +- Mark Elvers +source: +ignore: +--- + +Arthur mentioned that it would be great to know which packages build successfully with OxCaml and which don’t.
+ +With a little effort and PR#106, I was able to get opam-health-check to build OxCaml from the Jane Street branch and test the latest version of all the packages in opam.
+ +I created the switch using the branch janestreet/opam-repository#with-extensions. However, I ran into issues as autoconf isn’t included in the base images. I added an extra-command to install it, but found that these are executed last, after the switch has been created, and I needed autoconf before the switch was created. My PR moved the extra commands earlier in the build process.
Here is my config.yaml.
name: default
+port: 8080
+public-url: http://oxcaml.check.ci.dev
+admin-port: 9999
+auto-run-interval: 1680
+processes: 100
+enable-dune-cache: false
+enable-logs-compression: true
+default-repository: ocaml/opam-repository
+extra-repositories:
+- janestreet-with-extensions: janestreet/opam-repository#with-extensions
+with-test: false
+with-lower-bound: false
+list-command: opam list --available --installable --columns=package --short
+extra-command: sudo apt install autoconf -y
+platform:
+ os: linux
+ arch: x86_64
+ custom-pool:
+ distribution: debian-unstable
+ image: ocaml/opam:debian-12-ocaml-5.2@sha256:a17317e9abe385dc16b4390c64a374046d6dd562e80aea838d91c6c1335da357
+ocaml-switches:
+- 5.2.0+flambda2:
+ switch: 5.2.0+flambda2
+ build-with: opam
+This results in these commands, which build the switch for testing:
+ +sudo ln -f /usr/bin/opam-dev /usr/bin/opam
+rm -rf ~/opam-repository && git clone -q 'https://github.com/ocaml/opam-repository' ~/opam-repository && git -C ~/opam-repository checkout -q dbc9ec7b83bac3673185542221a571372b6abb35
+rm -rf ~/.opam && opam init -ya --bare --config ~/.opamrc-sandbox ~/opam-repository
+sudo apt install autoconf -y
+git clone -q 'https://github.com/janestreet/opam-repository' ~/'janestreet-with-extensions' && git -C ~/'janestreet-with-extensions' checkout -q 55a5d4c5e35a7365ddd6ffb3b87274a77f77deb5
+opam repository add --dont-select 'janestreet-with-extensions' ~/'janestreet-with-extensions'
+opam switch create --repositories=janestreet-with-extensions,default '5.2.0+flambda2' '5.2.0+flambda2'
+opam update --depexts
+The results are available at https://oxcaml.check.ci.dev.
diff --git a/data/planet/tunbury/opam-repo-ci-job-timeouts.md b/data/planet/tunbury/opam-repo-ci-job-timeouts.md new file mode 100644 index 0000000000..f289c7e870 --- /dev/null +++ b/data/planet/tunbury/opam-repo-ci-job-timeouts.md @@ -0,0 +1,112 @@ +--- +title: opam repo ci job timeouts +description: "It\u2019s Tuesday morning, and virtually all opam repo ci jobs are failing + with timeouts. This comes at a critical time as these are the first jobs following + the update of ocurrent/ocaml-version noted on 24th March." +url: https://www.tunbury.org/2025/04/04/opam-repo-ci/ +date: 2025-04-04T00:00:00-00:00 +preview_image: https://www.tunbury.org/images/opam.png +authors: +- Mark Elvers +source: +ignore: +--- + +It’s Tuesday morning, and virtually all opam repo ci jobs are failing with timeouts. This comes at a critical time as these are the first jobs following the update of ocurrent/ocaml-version noted on 24th March.
+ +The opam repo ci tests all PRs on opam-repository. The pipeline downloads Docker images, which contain the root filesystem for various Linux distributions, architectures, and OCaml versions, which are used as the base environment to run the tests. These base images are created by the base image builder. PR#317 update these base images in three ways:
+ +-
+
- Images for OCaml < 4.08 were removed. +
- The
opam-repository-archiveoverlay was removed as this contained the < 4.08 opam packages.
+ - The
ocaml-patches-overlayoverlay was removed as this was only needed to build OCaml < 4.08 on GCC 14.
+
Given these changes, I immediately assumed some element of these was the culprit.
+ +Here’s an example of a failure as reported in the log.
+ +2025-04-01 07:27.45 ---> using "9dd47386dd0565c83eac2e9d589d75bdd268a7f34f3c854d1db189e7a2e5f77b" from cache
+
+/: (user (uid 1000) (gid 1000))
+
+/: (workdir /home/opam)
+
+/home/opam: (run (shell "sudo ln -f /usr/bin/opam-dev /usr/bin/opam"))
+2025-04-01 07:27.45 ---> using "132d861be153666fd67b2e16b21c4de16e15e26f8d7d42f3bcddf0360ad147be" from cache
+
+/home/opam: (run (network host)
+ (shell "opam init --reinit --config .opamrc-sandbox -ni"))
+Configuring from /home/opam/.opamrc-sandbox, then /home/opam/.opamrc, and finally from built-in defaults.
+Checking for available remotes: rsync and local, git.
+ - you won't be able to use mercurial repositories unless you install the hg command on your system.
+ - you won't be able to use darcs repositories unless you install the darcs command on your system.
+
+This development version of opam requires an update to the layout of /home/opam/.opam from version 2.0 to version 2.2, which can't be reverted.
+You may want to back it up before going further.
+
+Continue? [Y/n] y
+[NOTE] The 'jobs' option was reset, its value was 39 and its new value will vary according to the current number of cores on your machine. You can restore the fixed value using:
+ opam option jobs=39 --global
+Format upgrade done.
+
+<><> Updating repositories ><><><><><><><><><><><><><><><><><><><><><><><><><><>
+2025-04-01 09:27.34: Cancelling: Timeout (120.0 minutes)
+Job cancelled
+2025-04-01 09:27.40: Timeout (120.0 minutes)
+With nearly all jobs taking 2 hours to run, the cluster was understandably backlogged!
+ +The issue could be reproduced with this Dockerfile:
+ +cd $(mktemp -d)
+git clone --recursive "https://github.com/ocaml/opam-repository.git" && cd "opam-repository" && git fetch origin "refs/pull/27696/head" && git reset --hard 46b8cc5a
+git fetch origin master
+git merge --no-edit 4d8fa0fb8fce3b6c8b06f29ebcfa844c292d4f3e
+cat > ../Dockerfile <<'END-OF-DOCKERFILE'
+FROM ocaml/opam:debian-12-ocaml-4.09@sha256:13bd7f0979922adb13049eecc387d65d7846a3058f7dd6509738933e88bc8d4a
+USER 1000:1000
+WORKDIR /home/opam
+RUN sudo ln -f /usr/bin/opam-dev /usr/bin/opam
+RUN opam init --reinit -ni
+RUN opam option solver=builtin-0install && opam config report
+ENV OPAMDOWNLOADJOBS="1"
+ENV OPAMERRLOGLEN="0"
+ENV OPAMPRECISETRACKING="1"
+ENV CI="true"
+ENV OPAM_REPO_CI="true"
+RUN rm -rf opam-repository/
+COPY --chown=1000:1000 . opam-repository/
+RUN opam repository set-url --strict default opam-repository/
+RUN opam update --depexts || true
+RUN opam pin add -k version -yn chrome-trace.3.18.0~alpha0 3.18.0~alpha0
+RUN opam reinstall chrome-trace.3.18.0~alpha0; \
+ res=$?; \
+ test "$res" != 31 && exit "$res"; \
+ export OPAMCLI=2.0; \
+ build_dir=$(opam var prefix)/.opam-switch/build; \
+ failed=$(ls "$build_dir"); \
+ partial_fails=""; \
+ for pkg in $failed; do \
+ if opam show -f x-ci-accept-failures: "$pkg" | grep -qF "\"debian-12\""; then \
+ echo "A package failed and has been disabled for CI using the 'x-ci-accept-failures' field."; \
+ fi; \
+ test "$pkg" != 'chrome-trace.3.18.0~alpha0' && partial_fails="$partial_fails $pkg"; \
+ done; \
+ test "${partial_fails}" != "" && echo "opam-repo-ci detected dependencies failing: ${partial_fails}"; \
+ exit 1
+
+END-OF-DOCKERFILE
+docker build -f ../Dockerfile .
+It was interesting to note which jobs still work. For example, builds on macOS and FreeBSD ran normally. This makes sense as these architectures don’t use the Docker base images. Looking further, opam repo ci attempts builds on opam 2.0, 2.1, 2.2, and 2.3 on Debian. These builds succeeded. Interesting. All the other builds use the latest version of opam built from the head of the master branch.
+ +Taking the failing Dockerfile above and replacing sudo ln -f /usr/bin/opam-dev /usr/bin/opam with sudo ln -f /usr/bin/opam-2.3 /usr/bin/opam immediately fixed the issue!
I pushed commit 7174953 to force opam repo ci to use opam 2.3 and opened issue#6448 on ocaml/opam. The working theory is that some change associated with PR#5892, which replace GNU patch with the OCaml patch library is the root cause.
+ +Musing on this issue with David, the idea of using the latest tag rather than head commit seemed like a good compromise. This allowed us to specifically test pre-release versions of opam when they were tagged but not be at the cutting edge with the risk of impacting a key service.
+ +We need the latest tag by version number, not by date, as we wouldn’t want to revert to testing on, for example, 2.1.7 if something caused a new release of the 2.1 series. The result was a function which runs git tag --format %(objectname) %(refname:strip=2) and semantically sorts the version numbers using OpamVersion.compare. See PR#318.
This is a high-level view of the steps required to update ocaml-repo-ci to use a new OCaml version.
+ +ocaml-repo-ci uses Docker images as the container’s root file system. The base image builder creates and maintains these images using ocurrent/ocaml-dockerfile. Both applications use the ocurrent/ocaml-version library as the definitive list of OCaml versions.
+ +1. Update ocurrent/ocaml-version
+ +Create a PR for changes to ocaml_version.ml with the details of the new release.
+ +2. Create and publish a new release of ocurrent/ocaml-version
Create the new release on GitHub and publish it to ocaml/opam-repository using opam, e.g.
opam publish --tag v4.0.1 https://github.com/ocurrent/ocaml-version/releases/download/v4.0.1/ocaml-version-4.0.1.tbz
+3. Update ocurrent/docker-base-images
+ +The change required is to update the opam repository SHA in the Dockerfile to pick up the latest version of ocurrent/ocaml-version.
+ +Run dune runtest --auto-promote to update the builds.expected file. Create a PR for these changes.
When the PR is pushed to the live branch ocurrent-deployer will pick up the change and deploy the new version.
4. Wait for the base images to build
+ +The base image builder refreshes the base images every seven days. Wait for the cycle to complete and the new images to be pushed to Docker Hub.
+ +5. Update ocurrent/opam-repo-ci
+ +Update the opam repository SHA in the Dockerfile. Update the doc/platforms.md and test/specs.expected using the following two commands.
+ +dune build @doc
+dune runtest --auto-promote
+Create a PR for this update. When the PR is pushed to the live branch ocurrent-deployer will pick up the change and deploy the new version.
This morning, Anil proposed that having an opam-repository that didn’t have old versions of the packages that require patches to work with OxCaml would be good.
+ +This is a fast-moving area, so this post is likely to be outdated very quickly, but at the time of writing, the development repository is https://github.com/janestreet/opam-repository#with-extensions. This is a fork of opam-repository but with some patched packages designated with +ox.
I have a short shell script which clones both opam-repository and https://github.com/janestreet/opam-repository#with-extensions and searches for all packages with +ox. All versions of these packages are removed from opam-repository and replaced with the single +ox version. The resulting repository is pushed to https://github.com/mtelvers/opam-repository-ox.
To test the repository (and show that eio doesn’t build), I have created a Dockerfile based largely on the base-image-builder format. This Dockerfile uses this modified opam-repository to build an OxCaml switch.
My build script and test Dockerfile are in [https://github.com/mtelvers/opam-repo-merge] (https://github.com/mtelvers/opam-repo-merge). Thanks to David for being the sounding board during the day!
diff --git a/data/planet/tunbury/opinion-is-it-time-to-stop-testing-with-opam-lt-22.md b/data/planet/tunbury/opinion-is-it-time-to-stop-testing-with-opam-lt-22.md new file mode 100644 index 0000000000..4cf30c11f3 --- /dev/null +++ b/data/planet/tunbury/opinion-is-it-time-to-stop-testing-with-opam-lt-22.md @@ -0,0 +1,67 @@ +--- +title: 'Opinion: Is it time to stop testing with opam < 2.2' +description: On the eve of the release of opam 2.4, is it time to stop testing with + opam < 2.2? +url: https://www.tunbury.org/2025/05/26/retire-legacy-opam/ +date: 2025-05-26T00:00:00-00:00 +preview_image: https://www.tunbury.org/images/opam.png +authors: +- Mark Elvers +source: +ignore: +--- + +On the eve of the release of opam 2.4, is it time to stop testing with opam < 2.2?
+ +Over the weekend, we have been seeing numerous failures across the ecosystem due to the unavailability of the camlcity.org. This website hosts the source for the findlib package. A typical error report is shown below:
#32 [build-opam-doc 5/14] RUN opam install odoc
+#32 258.6 [ERROR] Failed to get sources of ocamlfind.1.9.6: curl error code 504
+#32 258.6
+#32 258.6 #=== ERROR while fetching sources for ocamlfind.1.9.6 =========================#
+#32 258.6 OpamSolution.Fetch_fail("http://download.camlcity.org/download/findlib-1.9.6.tar.gz (curl: code 504 while downloading http://download.camlcity.org/download/findlib-1.9.6.tar.gz)")
+#32 259.0
+#32 259.0
+#32 259.0 <><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
+#32 259.0 +- The following actions failed
+#32 259.0 | - fetch ocamlfind 1.9.6
+#32 259.0 +-
+The most high-profile failure has been the inability to update opam.ocaml.org. See issue#172. This has also affected the deployment of ocaml.org.
+ +Late last year, Hannes proposed adding our archive mirror to the base image builder. issue#306. However, this requires opam 2.2 or later. We have long maintained that while supported distributions still package legacy versions, we should continue to test against these versions.
+ +The testing of the legacy versions is limited to opam-repo-ci testing on Debian 12 on AMD64 using a test matrix of OCaml 4.14 and 5.3 with each of opam 2.0, 2.1 and 2.2. These tests often fail to find a solution within the timeout. We have tried increasing the timeout by a factor of 10 to no avail. All of opam-repo-ci’s other tests use the current development version. OCaml-CI only tests using the current release version.
+ +[ERROR] Sorry, resolution of the request timed out.
+ Try to specify a simpler request, use a different solver, or increase the allowed time by setting OPAMSOLVERTIMEOUT to a bigger value (currently, it is set to 60.0 seconds).
+The base image default is opam 2.0, as ~/.opam can’t be downgraded; therefore, we can’t set a mirror archive flag in the base images.
A typical Dockerfile starts by replacing opam 2.0 with the latest version and reinitialising.
FROM ocaml/opam:debian-12-ocaml-4.14 AS build
+RUN sudo ln -sf /usr/bin/opam-2.3 /usr/bin/opam && opam init --reinit -ni
+...
+To include the archive mirror, we should add a follow-up of:
+ +RUN opam option --global 'archive-mirrors+="https://opam.ocaml.org/cache"'
+Dropping 2.0 and 2.1, and arguably 2.2 as well, from the base images would considerably decrease the time taken to build the base images, as opam is built from the source each week for each distribution/architecture.
+ +RUN git clone https://github.com/ocaml/opam /tmp/opam && cd /tmp/opam && cp -P -R -p . ../opam-sources && git checkout 4267ade09ac42c1bd0b84a5fa61af8ccdaadef48 && env MAKE='make -j' shell/bootstrap-ocaml.sh && make -C src_ext cache-archives
+RUN cd /tmp/opam-sources && cp -P -R -p . ../opam-build-2.0 && cd ../opam-build-2.0 && git fetch -q && git checkout adc1e1829a2bef5b240746df80341b508290fe3b && ln -s ../opam/src_ext/archives src_ext/archives && env PATH="/tmp/opam/bootstrap/ocaml/bin:$PATH" ./configure --enable-cold-check && env PATH="/tmp/opam/bootstrap/ocaml/bin:$PATH" make lib-ext all && mkdir -p /usr/bin && cp /tmp/opam-build-2.0/opam /usr/bin/opam-2.0 && chmod a+x /usr/bin/opam-2.0 && rm -rf /tmp/opam-build-2.0
+RUN cd /tmp/opam-sources && cp -P -R -p . ../opam-build-2.1 && cd ../opam-build-2.1 && git fetch -q && git checkout 263921263e1f745613e2882745114b7b08f3608b && ln -s ../opam/src_ext/archives src_ext/archives && env PATH="/tmp/opam/bootstrap/ocaml/bin:$PATH" ./configure --enable-cold-check --with-0install-solver && env PATH="/tmp/opam/bootstrap/ocaml/bin:$PATH" make lib-ext all && mkdir -p /usr/bin && cp /tmp/opam-build-2.1/opam /usr/bin/opam-2.1 && chmod a+x /usr/bin/opam-2.1 && rm -rf /tmp/opam-build-2.1
+RUN cd /tmp/opam-sources && cp -P -R -p . ../opam-build-2.2 && cd ../opam-build-2.2 && git fetch -q && git checkout 01e9a24a61e23e42d513b4b775d8c30c807439b2 && ln -s ../opam/src_ext/archives src_ext/archives && env PATH="/tmp/opam/bootstrap/ocaml/bin:$PATH" ./configure --enable-cold-check --with-0install-solver --with-vendored-deps && env PATH="/tmp/opam/bootstrap/ocaml/bin:$PATH" make lib-ext all && mkdir -p /usr/bin && cp /tmp/opam-build-2.2/opam /usr/bin/opam-2.2 && chmod a+x /usr/bin/opam-2.2 && rm -rf /tmp/opam-build-2.2
+RUN cd /tmp/opam-sources && cp -P -R -p . ../opam-build-2.3 && cd ../opam-build-2.3 && git fetch -q && git checkout 35acd0c5abc5e66cdbd5be16ba77aa6c33a4c724 && ln -s ../opam/src_ext/archives src_ext/archives && env PATH="/tmp/opam/bootstrap/ocaml/bin:$PATH" ./configure --enable-cold-check --with-0install-solver --with-vendored-deps && env PATH="/tmp/opam/bootstrap/ocaml/bin:$PATH" make lib-ext all && mkdir -p /usr/bin && cp /tmp/opam-build-2.3/opam /usr/bin/opam-2.3 && chmod a+x /usr/bin/opam-2.3 && rm -rf /tmp/opam-build-2.3
+RUN cd /tmp/opam-sources && cp -P -R -p . ../opam-build-master && cd ../opam-build-master && git fetch -q && git checkout 4267ade09ac42c1bd0b84a5fa61af8ccdaadef48 && ln -s ../opam/src_ext/archives src_ext/archives && env PATH="/tmp/opam/bootstrap/ocaml/bin:$PATH" ./configure --enable-cold-check --with-0install-solver --with-vendored-deps && env PATH="/tmp/opam/bootstrap/ocaml/bin:$PATH" make lib-ext all && mkdir -p /usr/bin && cp /tmp/opam-build-master/opam /usr/bin/opam-master && chmod a+x /usr/bin/opam-master && rm -rf /tmp/opam-build-master
+Furthermore, after changing the opam version, we must run opam init --reinit -ni, which is an expensive command. If the base images defaulted to the current version, we would have faster builds.
The final benefit, of course, would be that we could set the archive-mirror and reduce the number of transient failures due to network outages.
Would you like to build every package in opam in a single Dockerfile using BuildKit?
+ +In mtelvers/package-tool, I have combined various opam sorting and graphing functions into a CLI tool that will work on a checked-out opam-repository. Many of these flags can be combined.
+ +Package version
+ +package-tool --opam-repository ~/opam-repository <package>
+The package can be given as 0install.2.18 or 0install. The former specifies a specific version while the latter processes the latest version. --all-versions can be specified to generate files for all package versions.
Dependencies
+ +Dump the dependencies for the latest version of 0install into a JSON file.
+ +package-tool --opam-repository ~/opam-repository --deps 0install
+Produces 0install.2.18-deps.json:
{"yojson.3.0.0":["dune.3.19.1"],
+"xmlm.1.4.0":["topkg.1.0.8"],
+"topkg.1.0.8":["ocamlfind.1.9.8","ocamlbuild.0.16.1"],
+...
+"0install-solver.2.18"]}
+Installation order
+ +Create a list showing the installation order for the given package.
+ +package-tool --opam-repository ~/opam-repository --list 0install
+Produces 0install.2.18-list.json:
["ocaml-compiler.5.3.0",
+"ocaml-base-compiler.5.3.0",
+...
+"0install.2.18"]
+Solution DAG
+ +Output the solution graph in Graphviz format, which can then be converted into a PDF with dot.
package-tool --opam-repository ~/opam-repository --dot 0install
+dot -Tpdf 0install.2.18.dot 0install.2.18.pdf
+OCaml version
+ +By default, OCaml 5.3.0 is used, but this can be changed using the --ocaml 4.14.2 parameter.
Dockerfile
+ +The --dockerfile argument creates a Dockerfile to test the installation.
package-tool --opam-repository ~/opam-repository --dockerfile --all-versions 0install
+For example, the above command line outputs 5 Dockerfiles.
+ +-
+
- 0install.2.15.1.dockerfile +
- 0install.2.15.2.dockerfile +
- 0install.2.16.dockerfile +
- 0install.2.17.dockerfile +
- 0install.2.18.dockerfile +
As an example, 0install.2.18.dockerfile, contains:
FROM debian:12 AS builder_0install_2_18
+RUN apt update && apt upgrade -y
+RUN apt install -y build-essential git rsync unzip curl sudo
+RUN if getent passwd 1000; then userdel -r $(id -nu 1000); fi
+RUN adduser --uid 1000 --disabled-password --gecos '' opam
+ADD --chown=root:root --chmod=0755 [ "https://github.com/ocaml/opam/releases/download/2.3.0/opam-2.3.0-x86_64-linux", "/usr/local/bin/opam" ]
+RUN echo 'opam ALL=(ALL:ALL) NOPASSWD:ALL' >> /etc/sudoers.d/opam
+RUN chmod 440 /etc/sudoers.d/opam
+USER opam
+WORKDIR /home/opam
+ENV OPAMYES="1" OPAMCONFIRMLEVEL="unsafe-yes" OPAMERRLOGLEN="0" OPAMPRECISETRACKING="1"
+ADD --chown=opam:opam --keep-git-dir=false [ ".", "/home/opam/opam-repository" ]
+RUN opam init default -k local ~/opam-repository --disable-sandboxing --bare
+RUN opam switch create default --empty
+RUN opam install ocaml-compiler.5.3.0 >> build.log 2>&1 || echo 'FAILED' >> build.log
+RUN opam install ocaml-base-compiler.5.3.0 >> build.log 2>&1 || echo 'FAILED' >> build.log
+...
+RUN opam install 0install-solver.2.18 >> build.log 2>&1 || echo 'FAILED' >> build.log
+RUN opam install 0install.2.18 >> build.log 2>&1 || echo 'FAILED' >> build.log
+ENTRYPOINT [ "opam", "exec", "--" ]
+CMD bash
+This can be built using Docker in the normal way. Note that the build context is your checkout of opam-repository.
+ +docker build -f 0install.2.18.dockerfile ~/opam-repository
+Additionally, it outputs Dockerfile, which contains the individual package builds as a multistage build and an aggregation stage as the final layer:
FROM debian:12 AS results
+WORKDIR /results
+RUN apt update && apt upgrade -y
+RUN apt install -y less
+COPY --from=builder_0install_2_15_1 [ "/home/opam/build.log", "/results/0install.2.15.1" ]
+COPY --from=builder_0install_2_15_2 [ "/home/opam/build.log", "/results/0install.2.15.2" ]
+COPY --from=builder_0install_2_16 [ "/home/opam/build.log", "/results/0install.2.16" ]
+COPY --from=builder_0install_2_17 [ "/home/opam/build.log", "/results/0install.2.17" ]
+COPY --from=builder_0install_2_18 [ "/home/opam/build.log", "/results/0install.2.18" ]
+CMD bash
+Build all the versions of 0install in parallel using BuildKit’s layer caching:
+ +docker build -f Dockerfile -t opam-results ~/opam-repository
+We can inspect the build logs in the Docker container:
+ +$ docker run --rm -it opam-results
+root@b28da667e754:/results# ls^C
+root@b28da667e754:/results# ls -l
+total 76
+-rw-r--r-- 1 1000 1000 12055 Jul 22 20:17 0install.2.15.1
+-rw-r--r-- 1 1000 1000 15987 Jul 22 20:19 0install.2.15.2
+-rw-r--r-- 1 1000 1000 15977 Jul 22 20:19 0install.2.16
+-rw-r--r-- 1 1000 1000 16376 Jul 22 20:19 0install.2.17
+-rw-r--r-- 1 1000 1000 15150 Jul 22 20:19 0install.2.18
+Annoyingly, Docker doesn’t seem to be able to cope with all of opam at once. I get various RPC errors.
+ +[+] Building 2.9s (4/4) FINISHED docker:default
+ => [internal] load build definition from Dockerfile
+ => => transferring dockerfile: 10.79MB
+ => resolve image config for docker-image://docker.io/docker/dockerfile:1
+ => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:9857836c9ee4268391bb5b09f9f157f3c91bb15821bb77969642813b0d00518d
+ => [internal] load build definition from Dockerfile
+ERROR: failed to receive status: rpc error: code = Unavailable desc = error reading from server: connection error: COMPRESSION_ERROR
+It’s Pi Day 2025
+ +Archimedes calculated the perimeter of inscribed regular polygons +within a circle to approximate the value of π.
+ +A square inscribed in a unit circle can be divided into four right +triangles with two sides of unit length, corresponding to the radius of +the circle. The third side can be calculated by Pythagoras’ theorem to +be √2. The perimeter of the square would be 4√2. Given, C=πd, we +can calculate π from the circumference by dividing it by the diameter, +2, giving 2√2.
+ +
CA, CD and CB are all the unit radius. AB is √2 as calculated above. The +angle ACB can be bisected with the line CD. EB is half of AB. Using +Pythagoras’ theorem on the triangle BCE we can calculated CE. DE is then +1 - CE, allowing us to use Pythagoras’ theorem for a final time on BDE to +calculated BD. The improved approximation of the perimeter is now 8 x BD.
+ +We can iterate on this process using the following code:
+ +let rec pi edge_squared sides = function
+ | 0 -> sides *. Float.sqrt(edge_squared) /. 2.
+ | n ->
+ let edge_squared = 2. -. 2. *. Float.sqrt (1. -. edge_squared /. 4.) in
+ let sides = sides *. 2. in
+ pi edge_squared sides (n - 1)
+
+let approximation = pi 2. 4. 13
+let () = Printf.printf "pi %.31f\n" approximation
+I found this method quite interesting. Usually, as the number of +iterations increases the approximation of π becomes more accurate +with the delta between each step becoming smaller until the difference +is effectively zero (given the limited precision of the floating +calculation). However, in this case, after 13 iterations the +approximation becomes worse!
+ +| iteration | +approximation | +% error | +
|---|---|---|
| 0 | +2.8284271247461902909492437174777 | +9.968368 | +
| 1 | +3.0614674589207178101446515938733 | +2.550464 | +
| 2 | +3.1214451522580528575190328410827 | +0.641315 | +
| 3 | +3.1365484905459406483885231864406 | +0.160561 | +
| 4 | +3.1403311569547391890466769837076 | +0.040155 | +
| 5 | +3.1412772509327568926096319046337 | +0.010040 | +
| 6 | +3.1415138011441454679584239784162 | +0.002510 | +
| 7 | +3.1415729403678827047485810908256 | +0.000627 | +
| 8 | +3.1415877252799608854161306226160 | +0.000157 | +
| 9 | +3.1415914215046352175875199463917 | +0.000039 | +
| 10 | +3.1415923456110768086091411532834 | +0.000010 | +
| 11 | +3.1415925765450043449789063743083 | +0.000002 | +
| 12 | +3.1415926334632482408437681442592 | +0.000001 | +
| 13 | +3.1415926548075892021927302266704 | +-0.000000 | +
| 14 | +3.1415926453212152935634549066890 | +0.000000 | +
| 15 | +3.1415926073757196590463536267634 | +0.000001 | +
| 16 | +3.1415929109396727447744979144773 | +-0.000008 | +
| 17 | +3.1415941251951911006301543238806 | +-0.000047 | +
| 18 | +3.1415965537048196054570325941313 | +-0.000124 | +
| 19 | +3.1415965537048196054570325941313 | +-0.000124 | +
| 20 | +3.1416742650217575061333263874985 | +-0.002598 | +
| 21 | +3.1418296818892015309643284126651 | +-0.007545 | +
| 22 | +3.1424512724941338071005247911671 | +-0.027331 | +
| 23 | +3.1424512724941338071005247911671 | +-0.027331 | +
| 24 | +3.1622776601683795227870632515987 | +-0.658424 | +
| 25 | +3.1622776601683795227870632515987 | +-0.658424 | +
| 26 | +3.4641016151377543863532082468737 | +-10.265779 | +
| 27 | +4.0000000000000000000000000000000 | +-27.323954 | +
| 28 | +0.0000000000000000000000000000000 | +100.000000 | +
Using the decimal package +we can specify the floating point precision we want allowing us to +get to 100 decimal places in 165 steps.
+ +open Decimal
+
+let context = Context.make ~prec:200 ()
+let two = of_int 2
+let four = of_int 4
+
+let rec pi edge_squared sides n =
+ match n with
+ | 0 -> mul ~context sides (div ~context (sqrt ~context edge_squared) two)
+ | n ->
+ let edge_squared =
+ sub ~context two
+ (mul ~context two
+ (sqrt ~context (sub ~context one (div ~context edge_squared four))))
+ in
+ let sides = mul ~context sides two in
+ pi edge_squared sides (Int.pred n)
+
+let () = pi two four 165 |> to_string ~context |> Printf.printf "%s\n"
+This code is available on GitHub
diff --git a/data/planet/tunbury/playing-with-capn-proto.md b/data/planet/tunbury/playing-with-capn-proto.md new file mode 100644 index 0000000000..3ebe98eeb5 --- /dev/null +++ b/data/planet/tunbury/playing-with-capn-proto.md @@ -0,0 +1,112 @@ +--- +title: "Playing with Cap\u2019n Proto" +description: "Cap\u2019n Proto has become a hot topic recently and while this is used + for many OCaml-CI services, I spent some time creating a minimal application." +url: https://www.tunbury.org/2025/03/17/capnproto/ +date: 2025-03-17T00:00:00-00:00 +preview_image: https://www.tunbury.org/images/capnproto-logo.png +authors: +- Mark Elvers +source: +ignore: +--- + +Cap’n Proto has become a hot topic recently and while this is used for many OCaml-CI services, I spent some time creating a minimal application.
+ +Firstly create a schema with a single interface whch accepts a file name and returns the content.
+ +interface Foo {
+ get @0 (name :Text) -> (reply :Text);
+}
+This schema can then be compiled into the bindings for your required language. e.g. capnp compile -o ocaml:. schema.capnp
In practice this need not be done by hand as we can use a dune rule to do this.
(rule
+ (targets foo_api.ml foo_api.mli)
+ (deps foo_api.capnp)
+ (action (run capnp compile -o %{bin:capnpc-ocaml} %{deps})))
+On the server side we now need to extend the automatically generate code to actually implement the interface. This code is largely boilerplate.
+ +module Api = Foo_api.MakeRPC(Capnp_rpc)
+
+open Capnp_rpc.Std
+
+let read_from_file filename = In_channel.with_open_text filename @@ fun ic -> In_channel.input_all ic
+
+let local =
+ let module Foo = Api.Service.Foo in
+ Foo.local @@ object
+ inherit Foo.service
+
+ method get_impl params release_param_caps =
+ let open Foo.Get in
+ let name = Params.name_get params in
+ release_param_caps ();
+ let response, results = Service.Response.create Results.init_pointer in
+ Results.reply_set results (read_from_file name);
+ Service.return response
+ end
+The server needs to generate the capability file needed to access the service and wait for incoming connections.
+ +let cap_file = "echo.cap"
+
+let serve config =
+ Switch.run @@ fun sw ->
+ let service_id = Capnp_rpc_unix.Vat_config.derived_id config "main" in
+ let restore = Restorer.single service_id (Foo.local) in
+ let vat = Capnp_rpc_unix.serve ~sw ~restore config in
+ match Capnp_rpc_unix.Cap_file.save_service vat service_id cap_file with
+ | Error `Msg m -> failwith m
+ | Ok () ->
+ traceln "Server running. Connect using %S." cap_file;
+ Fiber.await_cancel ()
+The client application imports the capability file and calls the service Foo.get.
let run_client service =
+ let x = Foo.get service "client.ml" in
+ traceln "%S" x
+
+let connect net uri =
+ Switch.run @@ fun sw ->
+ let client_vat = Capnp_rpc_unix.client_only_vat ~sw net in
+ let sr = Capnp_rpc_unix.Vat.import_exn client_vat uri in
+ Capnp_rpc_unix.with_cap_exn sr run_client
+Where Foo.get is defined like this
module Foo = Api.Client.Foo
+
+let get t name =
+ let open Foo.Get in
+ let request, params = Capability.Request.create Params.init_pointer in
+ Params.name_set params name;
+ Capability.call_for_value_exn t method_id request |> Results.reply_get
+Run the server application passing it parameters of where to save the private key and which interface/port to listen on.
+ +$ dune exec -- ./server.exe --capnp-secret-key-file ./server.pem --capnp-listen-address tcp:127.0.0.1:7000
++Server running. Connect using "echo.cap".
+The .cap looks like this
capnp://sha-256:f5BAo2n_2gVxUdkyzYsIuitpA1YT_7xFg31FIdNKVls@127.0.0.1:7000/6v45oIvGQ6noMaLOh5GHAJnGJPWEO5A3Qkt0Egke4Ic
+In another window, invoke the client.
+ +$ dune exec -- ./client.exe ./echo.cap
+The full code is available on Github.
diff --git a/data/planet/tunbury/posthog-on-ocamlorg.md b/data/planet/tunbury/posthog-on-ocamlorg.md new file mode 100644 index 0000000000..16c6416043 --- /dev/null +++ b/data/planet/tunbury/posthog-on-ocamlorg.md @@ -0,0 +1,30 @@ +--- +title: Posthog on OCaml.org +description: Sabine would like to switch OCaml.org from using Plausible over to Posthog. + The underlying reason for the move is that the self-hosted product from Posthog + has more features than the equivalent from Plausible. Of particular interest is + the heatmap feature to assess the number of visitors who finish the Tour of OCaml. +url: https://www.tunbury.org/2025/05/12/posthog/ +date: 2025-05-12T12:00:00-00:00 +preview_image: https://www.tunbury.org/images/posthog.png +authors: +- Mark Elvers +source: +ignore: +--- + +Sabine would like to switch OCaml.org from using Plausible over to Posthog. The underlying reason for the move is that the self-hosted product from Posthog has more features than the equivalent from Plausible. Of particular interest is the heatmap feature to assess the number of visitors who finish the Tour of OCaml.
+ +Posthog has documentation on the self-hosted solution. In short, create a VM with 4 vCPU, 16GB RAM, and 30GB storage and run the setup script:
+ +/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/posthog/posthog/HEAD/bin/deploy-hobby)”
+Any subsequent upgrades can be achieved with:
+ +/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/posthog/posthog/HEAD/bin/upgrade-hobby)"
+After installation, I created a public dashboard as with Plausible. I also enabled the option Discard client IP data.
The OCaml website can be updated with PR#3101.
diff --git a/data/planet/tunbury/q2-summary.md b/data/planet/tunbury/q2-summary.md new file mode 100644 index 0000000000..3ae661069d --- /dev/null +++ b/data/planet/tunbury/q2-summary.md @@ -0,0 +1,56 @@ +--- +title: Q2 Summary +description: "I am grateful for Tarides\u2019 sponsorship of my OCaml work. Below + is a summary of my activities in Q2 2025." +url: https://www.tunbury.org/2025/07/18/q2-summary/ +date: 2025-07-18T12:00:00-00:00 +preview_image: https://www.tunbury.org/images/tarides.png +authors: +- Mark Elvers +source: +ignore: +--- + +I am grateful for Tarides’ sponsorship of my OCaml work. Below is a summary of my activities in Q2 2025.
+ +OCaml Infrastructure and Development
+ +OCaml Maintenance Activities
+ +General maintenance work on OCaml’s infrastructure spanned many areas, including updating minimum supported OCaml versions from 4.02 to 4.08 and addressing issues with opam-repo-ci job timeouts. Platform-specific work included resolving compatibility issues with Fedora 42 and GCC 15, addressing Ubuntu AppArmor conflicts affecting runc operations, and managing macOS Sequoia upgrades across the Mac Mini CI workers. Complex build issues were investigated and resolved, including C++ header path problems in macOS workers and FreeBSD system upgrades for the CI infrastructure.
+ +OCaml Infrastructure Migration
+ +Due to the impending sunset of the Equinix Metal platform, the OCaml community services needed to be migrated. Services including OCaml-CI, opam-repo-ci, and the opam.ocaml.org deployment pipeline were migrated to new blade servers. The migration work was planned to minimise service disruption, which was kept to just a few minutes. Complete procedures were documented, including Docker volume transfers and rsync strategies.
+ +opam2web Deployment
+ +Optimisation work was undertaken on the deployment pipeline for opam2web, which powers opam.ocaml.org, to address the more than two-hour deployment time. The primary issue was the enormous size of the opam2web Docker image, which exceeded 25GB due to the inclusion of complete opam package archives. The archive was moved to a separate layer, allowing Docker to cache the layer and reducing the deployment time to 20 minutes.
+ +opam Dependency Graphs
+ +Algorithms for managing OCaml package dependencies were investigated, including topological sorting to determine the optimal package installation order. This work extended to handling complex dependency scenarios, including post-dependencies and optional dependencies. Implemented a transitive reduction algorithm to create a dependency graph with minimal edge counts while preserving the same dependency relationships, enabling more efficient package management and installation processes.
+ +OCaml Developments under Windows
+ +Significant work was undertaken to bring containerization technologies to OCaml development on Windows. This included implementing a tool to create host compute networks via the Windows API, tackling limitations with NTFS hard links, and implementing copy-on-write reflink tool for Windows.
+ +OxCaml Support
+ +Support for the new OxCaml compiler variant included establishing an opam repository and testing which existing OCaml packages successfully built with the new compiler.
+ +ZFS Storage and Hardware Deployment
+ +Early in the quarter, a hardware deployment project centred around Dell PowerEdge R640 servers with a large-scale SSD storage was undertaken. The project involved deploying multiple batches of Kingston 7.68TB SSD drives, creating automated deployments for Ubuntu using network booting with EFI and cloud-init configuration. Experimented with ZFS implementation as a root filesystem, which was possibly but ultimately discarded and explored dm-cache for SSD acceleration of spinning disk arrays. Investigated using ZFS as a distributed storage archive system using an Ansible-based deployment strategy based upon a YAML description.
+ +Talos II Repairs
+ +Significant hardware reliability issues affected two Raptor Computing Talos II POWER9 machines. The first system experienced complete lockups after as little as 20 minutes of operation, while the second began exhibiting similar problems requiring daily power cycling. Working with Raptor Computing support to isolate the issues, upgrading firmware and eventually swapping CPUs between the systems resolved the issue. Concurrently, this provided an opportunity to analyse the performance of OBuilder operations on POWER9 systems, comparing OverlayFS on TMPFS versus BTRFS on NVMe storage, resulting in optimised build performance.
+ +EEG Systems Investigations
+ +Various software solutions and research platforms were explored as part of a broader system evaluation. This included investigating Slurm Workload Manager for compute resource scheduling, examining Gluster distributed filesystem capabilities, and implementing Otter Wiki with Raven authentication integration for collaborative documentation. Research extended to modern research data management platforms, exploring InvenioRDM for scientific data archival and BON in a Box for biodiversity analysis workflows. To support the Teserra workshop, a multi-user Jupyter environment was set up using Docker containerization.
+ +Miscellaneous Technical Explorations
+ +Diverse technical explorations included implementing Bluesky Personal Data Server and developing innovative SSH authentication mechanisms using the ATProto network by extracting SSH public keys from Bluesky profiles. Additional projects included developing OCaml-based API tools for Box cloud storage, creating Real Time Trains API integrations, and exploring various file synchronisation and backup solutions. Investigation of reflink copy mechanisms for efficient file operations using OCaml multicore.
diff --git a/data/planet/tunbury/real-time-trains-api.md b/data/planet/tunbury/real-time-trains-api.md new file mode 100644 index 0000000000..dc738f98bd --- /dev/null +++ b/data/planet/tunbury/real-time-trains-api.md @@ -0,0 +1,91 @@ +--- +title: Real Time Trains API +description: After the Heathrow substation electrical fire, I found myself in Manchester + with a long train ride ahead. Checking on Real Time Trains for the schedule I noticed + that they had an API. With time to spare, I registered for an account and downloaded + the sample code from ocaml-cohttp. +url: https://www.tunbury.org/2025/03/23/real-time-trains/ +date: 2025-03-23T00:00:00-00:00 +preview_image: https://www.tunbury.org/images/rtt.png +authors: +- Mark Elvers +source: +ignore: +--- + +After the Heathrow substation electrical fire, I found myself in Manchester with a long train ride ahead. Checking on Real Time Trains for the schedule I noticed that they had an API. With time to spare, I registered for an account and downloaded the sample code from ocaml-cohttp.
+ +The API account details uses HTTP basic authentication which is added via the HTTP header:
+ + let headers = Cohttp.Header.init () in
+ let headers =
+ Cohttp.Header.add_authorization headers (`Basic (user, password))
+The response from the API can be converted to JSON using Yojson.
+ +let json =
+ Eio.Buf_read.(parse_exn take_all) body ~max_size:max_int
+ |> Yojson.Safe.from_string
+The JSON field can be read using the Util functions. For example, Yojson.Basic.Util.member "services" json will read the services entry. Elements can be converted to lists with Yojson.Basic.Util.to_list. After a bit of hacking this turned out to be quite tedious to code.
As an alternative, I decided to use ppx_deriving_yojson.runtime. I described the JSON blocks as OCaml types, e.g. station as below.
type station = {
+ tiploc : string;
+ description : string;
+ workingTime : string;
+ publicTime : string;
+}
+[@@deriving yojson]
+The preprocessor automatically generates two functions:station_of_json and station_to_json which handle the conversion.
The only negative on this approach is that RTT doesn’t emit empty JSON fields, so they need to be flagged as possibly missing and a default value provided. For example, realtimeArrivalNextDay is not emitted unless the value is true.
realtimeArrivalNextDay : (bool[@default false]);
+Now once the JSON has been received we can just convert it to OCaml types very easily:
+ + match reply_of_yojson json with
+ | Ok reply ->
+ (* Use reply.services *)
+ | Error err -> Printf.printf "Error %s\n" err
+My work in progress code is available on GitHub
+ +dune exec --release -- rtt --user USER --pass PASS --station RTR
+rtt: [DEBUG] received 3923 bytes of body
+rtt: [DEBUG] received 4096 bytes of body
+rtt: [DEBUG] received 4096 bytes of body
+rtt: [DEBUG] received 4096 bytes of body
+rtt: [DEBUG] received 1236 bytes of body
+rtt: [DEBUG] end of inbound body
+2025-03-23 2132 W16178 1C69 1 Ramsgate St Pancras International
+2025-03-23 2132 W25888 9P59 2 Plumstead Rainham (Kent)
+2025-03-23 2136 J00119 1U28 2 London Victoria Ramsgate
+2025-03-23 2144 W25927 9P86 1 Rainham (Kent) Plumstead
+2025-03-23 2157 W16899 1C66 2 St Pancras International Ramsgate
+2025-03-23 2202 W25894 9P61 2 Plumstead Rainham (Kent)
+2025-03-23 2210 J26398 1U80 1 Ramsgate London Victoria
+2025-03-23 2214 W25916 9P70 1 Rainham (Kent) Plumstead
+2025-03-23 2232 W16910 1C73 1 Ramsgate St Pancras International
+2025-03-23 2232 W25900 9P63 2 Plumstead Rainham (Kent)
+2025-03-23 2236 J00121 1U30 2 London Victoria Ramsgate
+2025-03-23 2244 W25277 9A92 1 Rainham (Kent) Dartford
+2025-03-23 2257 W16450 1F70 2 St Pancras International Faversham
+2025-03-23 2302 W25906 9P65 2 Plumstead Rainham (Kent)
+2025-03-23 2314 W25283 9A94 1 Rainham (Kent) Dartford
+2025-03-23 2318 J00155 1U82 1 Ramsgate London Victoria
+2025-03-23 2332 W25912 9P67 2 Plumstead Gillingham (Kent)
+2025-03-23 2336 J00123 1U32 2 London Victoria Ramsgate
+2025-03-23 2344 W25289 9A96 1 Rainham (Kent) Dartford
+2025-03-23 2357 W16475 1F74 2 St Pancras International Faversham
+2025-03-23 0002 W25915 9P69 2 Plumstead Gillingham (Kent)
+2025-03-23 0041 J26381 1Z34 2 London Victoria Faversham
+Following my post on discuss.ocaml.org, I have created a new release of ocurrent/ocaml-version that moves the minimum version of OCaml, considered as recent, from 4.02 to 4.08.
+ +let recent = [ v4_08; v4_09; v4_10; v4_11; v4_12; v4_13; v4_14; v5_0; v5_1; v5_2; v5_3 ]
+This may feel like a mundane change, but OCaml-CI, opam-repo-ci, Docker base image builder among other things, use this to determine the set of versions of OCaml to test against. Therefore, as these services are updated, testing on the old releases will be removed.
diff --git a/data/planet/tunbury/reflink-copy.md b/data/planet/tunbury/reflink-copy.md new file mode 100644 index 0000000000..481e1fd996 --- /dev/null +++ b/data/planet/tunbury/reflink-copy.md @@ -0,0 +1,122 @@ +--- +title: Reflink Copy +description: "I hadn\u2019t intended to write another post about traversing a directory + structure or even thinking about it again, but weirdly, it just kept coming up again!" +url: https://www.tunbury.org/2025/07/15/reflink-copy/ +date: 2025-07-15T00:00:00-00:00 +preview_image: https://www.tunbury.org/images/ocaml-logo.png +authors: +- Mark Elvers +source: +ignore: +--- + +I hadn’t intended to write another post about traversing a directory structure or even thinking about it again, but weirdly, it just kept coming up again!
+ +Firstly, Patrick mentioned Eio.Path.read_dir and Anil mentioned bfs. Then Becky commented about XFS reflink performance, and I commented that the single-threaded nature of cp -r --reflink=always was probably hurting our obuilder performance tests.
Obuilder is written in LWT, which has Lwt_unix.readdir. What if we had a pool of threads that would traverse the directory structure in parallel and create a reflinked copy?
Creating a reflink couldn’t be easier. There’s an ioctl call that just does it. Such a contrast to the ReFS copy-on-write implementation on Windows!
#include <caml/mlvalues.h>
+#include <caml/memory.h>
+#include <caml/unixsupport.h>
+#include <sys/ioctl.h>
+#include <errno.h>
+
+#ifndef FICLONE
+#define FICLONE 0x40049409
+#endif
+
+value caml_ioctl_ficlone(value dst_fd, value src_fd) {
+ CAMLparam2(dst_fd, src_fd);
+ int result;
+
+ result = ioctl(Int_val(dst_fd), FICLONE, Int_val(src_fd));
+
+ if (result == -1) {
+ uerror("ioctl_ficlone", Nothing);
+ }
+
+ CAMLreturn(Val_int(result));
+}
+We can write a reflink copy function as shown below. (Excuse my error handling.) Interestingly, points to note: the permissions set via Unix.openfile are filtered through umask, and you need to Unix.fchown before Unix.fchmod if you want to set the suid bit set.
external ioctl_ficlone : Unix.file_descr -> Unix.file_descr -> int = "caml_ioctl_ficlone"
+
+let copy_file src dst stat =
+ let src_fd = Unix.openfile src [O_RDONLY] 0 in
+ let dst_fd = Unix.openfile dst [O_WRONLY; O_CREAT; O_TRUNC] 0o600 in
+ let _ = ioctl_ficlone dst_fd src_fd in
+ Unix.fchown dst_fd stat.st_uid stat.st_gid;
+ Unix.fchmod dst_fd stat.st_perm;
+ Unix.close src_fd;
+ Unix.close dst_fd;
+My LWT code created a list of all the files in a directory and then processed the list with Lwt_list.map_s (serially), returning promises for all the file operations and creating threads for new directory operations up to a defined maximum (8). If there was no thread capacity, it just recursed in the current thread. Copying a root filesystem, this gave me threads for var, usr, etc, just as we’d want. Wow! This was slow. Nearly 4 minutes to reflink 1.7GB!
What about using the threads library rather than LWT threads? This appears significantly better, bringing the execution time down to 40 seconds. However, I think a lot of that was down to my (bad) LWT implementation vs my somewhat better threads implementation.
+ +At this point, I should probably note that cp -r --reflink always on 1.7GB, 116,000 files takes 8.5 seconds on my machine using a loopback XFS. A sequential OCaml version, without the overhead of threads or any need to maintain a list of work to do, takes 9.0 seconds.
Giving up and getting on with other things was very tempting, but there was that nagging feeling of not bottoming out the problem.
+ +Using OCaml Multicore, we can write a true multi-threaded version. I took a slightly different approach, having a work queue of directories to process, and N worker threads taking work from the queue.
+ +Main Process: Starts with root directory
+ ↓
+WorkQueue: [process_dir(/root)]
+ ↓
+Domain 1: Takes work → processes files → adds subdirs to queue
+Domain 2: Takes work → processes files → adds subdirs to queue
+Domain 3: Takes work → processes files → adds subdirs to queue
+ ↓
+WorkQueue: [process_dir(/root/usr), process_dir(/root/var), ...]
+Below is a table showing the performance when using multiple threads compared to the baseline operation of cp and a sequential copy in OCaml.
| Copy command | +Duration (sec) | +
|---|---|
| cp -r –reflink=always | +8.49 | +
| Sequential | +8.80 | +
| 2 domains | +5.45 | +
| 4 domains | +3.28 | +
| 6 domains | +3.43 | +
| 8 domains | +5.24 | +
| 10 domains | +9.07 | +
The code is available on GitHub in mtelvers/reflink.
diff --git a/data/planet/tunbury/refs-eeg-intern-and-monteverde.md b/data/planet/tunbury/refs-eeg-intern-and-monteverde.md new file mode 100644 index 0000000000..51bfcf0400 --- /dev/null +++ b/data/planet/tunbury/refs-eeg-intern-and-monteverde.md @@ -0,0 +1,91 @@ +--- +title: ReFS, EEG Intern and Monteverde +description: In addition to the post from last week covering BON in a Box and OCaml + Functors, below are some additional notes. +url: https://www.tunbury.org/2025/07/07/refs-monteverde/ +date: 2025-07-07T00:00:00-00:00 +preview_image: https://www.tunbury.org/images/refs.png +authors: +- Mark Elvers +source: +ignore: +--- + +In addition to the post from last week covering BON in a Box and OCaml Functors, below are some additional notes.
+ +Resilient File System, ReFS
+ +I have previously stated that ReFS supports 1 million hard links per file; however, this is not the case. The maximum is considerably lower at 8191. That’s eight times more than NTFS, but still not very many.
+ +PS D:\> touch foo
+PS D:\> foreach ($i in 1..8192) {
+>> New-Item -ItemType HardLink -Path "foo-$i" -Target "foo"
+>> }
+
+
+ Directory: D:\
+
+
+Mode LastWriteTime Length Name
+---- ------------- ------ ----
+-a---- 07/07/2025 01:00 0 foo-1
+-a---- 07/07/2025 01:00 0 foo-2
+-a---- 07/07/2025 01:00 0 foo-3
+-a---- 07/07/2025 01:00 0 foo-4
+...
+-a---- 07/07/2025 01:00 0 foo-8190
+-a---- 07/07/2025 01:00 0 foo-8191
+New-Item : An attempt was made to create more links on a file than the file system supports
+At line:2 char:5
++ New-Item -ItemType HardLink -Path "foo-$i" -Target "foo"
++ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ + CategoryInfo : NotSpecified: (:) [New-Item], Win32Exception
+ + FullyQualifiedErrorId : System.ComponentModel.Win32Exception,Microsoft.PowerShell.Commands.NewItemCommand
+I had also investigated ReFS block cloning, which removed the requirement to create hard links, and wrote a ReFS-clone tool for Windows Server 2022. This works well until containerd is used to bind mount a directory on the volume. Once this has happened, attempts to create a block clone fail. To exclude my code as the root cause, I have tried Windows Server 2025, where commands such as copy and robocopy automatically perform block clones. Block cloning can be restored by rebooting the machine. I note that restarting containerd is not sufficient.
Removing files and folders on ReFS is impressively fast; however, this comes at a cost: freeing the blocks is a background activity that may take some time to be scheduled.
+ +File system performance with a focus on ZFS
+ +Several EEG interns started last week with this project under my supervision. In brief, we will examine file system performance on the filesystems supported by OBuilder before conducting more detailed investigations into factors affecting ZFS performance.
+ +Monteverde
+ +monteverde.cl.cam.ac.uk, has been installed in the rack. It has two AMD EPYC 9965 192-Core Processors, giving a total of 384 cores and 768 threads and 3TB of RAM.
+ +
From the logs, there are still some teething issues:
+ +[130451.620482] Large kmem_alloc(98304, 0x1000), please file an issue at:
+ https://github.com/openzfs/zfs/issues/new
+[130451.620486] CPU: 51 UID: 0 PID: 8594 Comm: txg_sync Tainted: P O 6.14.0-23-generic #23-Ubuntu
+[130451.620488] Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
+[130451.620489] Hardware name: Dell Inc. PowerEdge R7725/0KRFPX, BIOS 1.1.3 02/25/2025
+[130451.620490] Call Trace:
+[130451.620490] <TASK>
+[130451.620492] show_stack+0x49/0x60
+[130451.620493] dump_stack_lvl+0x5f/0x90
+[130451.620495] dump_stack+0x10/0x18
+[130451.620497] spl_kmem_alloc_impl.cold+0x17/0x1c [spl]
+[130451.620503] spl_kmem_zalloc+0x19/0x30 [spl]
+[130451.620508] multilist_create_impl+0x3f/0xc0 [zfs]
+[130451.620586] multilist_create+0x31/0x50 [zfs]
+[130451.620650] dmu_objset_sync+0x4c4/0x4d0 [zfs]
+[130451.620741] dsl_pool_sync_mos+0x34/0xc0 [zfs]
+[130451.620832] dsl_pool_sync+0x3c1/0x420 [zfs]
+[130451.620910] spa_sync_iterate_to_convergence+0xda/0x220 [zfs]
+[130451.620990] spa_sync+0x333/0x660 [zfs]
+[130451.621056] txg_sync_thread+0x1f5/0x270 [zfs]
+[130451.621137] ? __pfx_txg_sync_thread+0x10/0x10 [zfs]
+[130451.621207] ? __pfx_thread_generic_wrapper+0x10/0x10 [spl]
+[130451.621213] thread_generic_wrapper+0x5b/0x70 [spl]
+[130451.621217] kthread+0xf9/0x230
+[130451.621219] ? __pfx_kthread+0x10/0x10
+[130451.621221] ret_from_fork+0x44/0x70
+[130451.621223] ? __pfx_kthread+0x10/0x10
+[130451.621224] ret_from_fork_asm+0x1a/0x30
+[130451.621226] </TASK>
+Shuffling a list into a random order is usually handled by the Fisher-Yates Shuffle.
+ +It could be efficiently written in OCaml using arrays:
+ +Random.self_init ();
+
+let fisher_yates_shuffle arr =
+ let n = Array.length arr in
+ for i = n - 1 downto 1 do
+ let j = Random.int (i + 1) in
+ let temp = arr.(i) in
+ arr.(i) <- arr.(j);
+ arr.(j) <- temp
+ done
+However, I had a one-off requirement to randomise a list, and this approach felt very functional.
+ +Random.self_init ();
+
+let shuffle lst =
+ List.map (fun x -> (Random.bits (), x)) lst |> List.sort compare |> List.map snd
+Most of the time, you don’t think about how your file is linked. We’ve come to love dynamically linked files with their small file sizes and reduced memory requirements, but there are times when the convenience of a single binary download from a GitHub release page is really what you need.
+ +To do this in OCaml, we need to add -ccopt -static to the ocamlopt. I’m building with dune, so I can configure that in my dune file using a flags directive.
(flags (:standard -ccopt -static))
+This can be extended for maximum compatibility by additionally adding -ccopt -march=x86-64, which ensures the generated code will run on any x86_64 processor and will not use newer instruction set extensions like SSE3, AVX, etc.
So what about Windows? The Mingw tool chain accepts -static. Including (flags (:standard -ccopt "-link -Wl,-static -v")) got my options applied to my dune build:
x86_64-w64-mingw32-gcc -mconsole -L. -I"C:/Users/Administrator/my-app/_opam/lib/ocaml" -I"C:\Users\Administrator\my-app\_opam\lib\mccs" -I"C:\Users\Administrator\my-app\_opam\lib\mccs\glpk/internal" -I"C:\Users\Administrator\my-app\_opam\lib\opam-core" -I"C:\Users\Administrator\my-app\_opam\lib\sha" -I"C:/Users/Administrator/my-app/_opam/lib/ocaml\flexdll" -L"C:/Users/Administrator/my-app/_opam/lib/ocaml" -L"C:\Users\Administrator\my-app\_opam\lib\mccs" -L"C:\Users\Administrator\my-app\_opam\lib\mccs\glpk/internal" -L"C:\Users\Administrator\my-app\_opam\lib\opam-core" -L"C:\Users\Administrator\my-app\_opam\lib\sha" -L"C:/Users/Administrator/my-app/_opam/lib/ocaml\flexdll" -o "bin/main.exe" "C:\Users\ADMINI~1\AppData\Local\Temp\2\build_d62d04_dune\dyndllb7e0e8.o" "@C:\Users\ADMINI~1\AppData\Local\Temp\2\build_d62d04_dune\camlrespec7816" "-municode" "-Wl,-static"
+However, ldd showed that this wasn’t working:
$ ldd main.exe | grep mingw
+ libstdc++-6.dll => /mingw64/bin/libstdc++-6.dll (0x7ffabf3e0000)
+ libgcc_s_seh-1.dll => /mingw64/bin/libgcc_s_seh-1.dll (0x7ffac3130000)
+ libwinpthread-1.dll => /mingw64/bin/libwinpthread-1.dll (0x7ffac4b40000)
+I tried a lot of different variations. I asked Claude… then I asked @dra27 who recalled @kit-ty-kate working on this for opam. PR#5680
+ +The issue is the auto-response file, which precedes my static option. We can remove that by adding -noautolink, but now we must do all the work by hand and build a massive command line.
(executable
+ (public_name main)
+ (name main)
+ (flags (:standard -noautolink -cclib -lunixnat -cclib -lmccs_stubs -cclib -lmccs_glpk_stubs -cclib -lsha_stubs -cclib -lopam_core_stubs -cclib -l:libstdc++.a -cclib -l:libpthread.a -cclib -Wl,-static -cclib -ladvapi32 -cclib -lgdi32 -cclib -luser32 -cclib -lshell32 -cclib -lole32 -cclib -luuid -cclib -luserenv -cclib -lwindowsapp))
+ (libraries opam-client))
+It works, but it’s not for the faint-hearted.
+ +I additionally added (enabled_if (= %{os_type} Win32)) to my rule so it only runs on Windows.
@mseri raised issue #175 as the macOS workers cannot find the most basic C++ headers. I easily eliminated Obuilder, as opam install mccs.1.1+19 didn’t work on the macOS workers natively.
On face value, the problem appears pretty common, and there are numerous threads on Stack Overflow such as this one, however, the resolutions I tried didn’t work. I was reluctant to try some of the more intrusive changes like creating a symlink of every header from /usr/include/ to /Library/Developer/CommandLineTools/usr/include/c++/v1 as this doesn’t seem to be what Apple intends.
For the record, a program such as this:
+ +#include <iostream>
+
+using namespace std;
+
+int main() {
+ cout << "Hello World!" << endl;
+ return 0;
+}
+Fails like this:
+ +% c++ hello.cpp -o hello -v
+Apple clang version 17.0.0 (clang-1700.0.13.3)
+Target: x86_64-apple-darwin24.5.0
+Thread model: posix
+InstalledDir: /Library/Developer/CommandLineTools/usr/bin
+ "/Library/Developer/CommandLineTools/usr/bin/clang" -cc1 -triple x86_64-apple-macosx15.0.0 -Wundef-prefix=TARGET_OS_ -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -Werror=implicit-function-declaration -emit-obj -dumpdir hello- -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name hello.cpp -mrelocation-model pic -pic-level 2 -mframe-pointer=all -fno-strict-return -ffp-contract=on -fno-rounding-math -funwind-tables=2 -target-sdk-version=15.4 -fvisibility-inlines-hidden-static-local-var -fdefine-target-os-macros -fno-assume-unique-vtables -fno-modulemap-allow-subdirectory-search -target-cpu penryn -tune-cpu generic -debugger-tuning=lldb -fdebug-compilation-dir=/Users/administrator/x -target-linker-version 1167.4.1 -v -fcoverage-compilation-dir=/Users/administrator/x -resource-dir /Library/Developer/CommandLineTools/usr/lib/clang/17 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -internal-isystem /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1 -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include -internal-isystem /Library/Developer/CommandLineTools/usr/lib/clang/17/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -internal-externc-isystem /Library/Developer/CommandLineTools/usr/include -Wno-reorder-init-list -Wno-implicit-int-float-conversion -Wno-c99-designator -Wno-final-dtor-non-final-class -Wno-extra-semi-stmt -Wno-misleading-indentation -Wno-quoted-include-in-framework-header -Wno-implicit-fallthrough -Wno-enum-enum-conversion -Wno-enum-float-conversion -Wno-elaborated-enum-base -Wno-reserved-identifier -Wno-gnu-folding-constant -fdeprecated-macro -ferror-limit 19 -stack-protector 1 -fstack-check -mdarwin-stkchk-strong-link -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fno-cxx-modules -fskip-odr-check-in-gmf -fcxx-exceptions -fexceptions -fmax-type-align=16 -fcommon -fcolor-diagnostics -clang-vendor-feature=+disableNonDependentMemberExprInCurrentInstantiation -fno-odr-hash-protocols -clang-vendor-feature=+enableAggressiveVLAFolding -clang-vendor-feature=+revert09abecef7bbf -clang-vendor-feature=+thisNoAlignAttr -clang-vendor-feature=+thisNoNullAttr -clang-vendor-feature=+disableAtImportPrivateFrameworkInImplementationError -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /var/folders/sh/9c8b7hzd2wb1g2_ky78vqw5r0000gn/T/hello-a268ab.o -x c++ hello.cpp
+clang -cc1 version 17.0.0 (clang-1700.0.13.3) default target x86_64-apple-darwin24.5.0
+ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include"
+ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/SubFrameworks"
+ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/Library/Frameworks"
+#include "..." search starts here:
+#include <...> search starts here:
+ /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1
+ /Library/Developer/CommandLineTools/usr/lib/clang/17/include
+ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
+ /Library/Developer/CommandLineTools/usr/include
+ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)
+End of search list.
+hello.cpp:1:10: fatal error: 'iostream' file not found
+ 1 | #include <iostream>
+ | ^~~~~~~~~~
+1 error generated.
+That first folder looked strange: bin/../include/c++/v1. Really? What’s in there? Not much:
% ls -l /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1
+total 40
+-rw-r--r-- 1 root wheel 44544 7 Apr 2022 __functional_03
+-rw-r--r-- 1 root wheel 6532 7 Apr 2022 __functional_base_03
+-rw-r--r-- 1 root wheel 2552 7 Apr 2022 __sso_allocator
+I definitely have iostream on the machine:
% ls -l /Library/Developer/CommandLineTools/SDKs/MacOSX*.sdk/usr/include/c++/v1/iostream
+-rw-r--r-- 1 root wheel 1507 8 Mar 03:36 /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/iostream
+-rw-r--r-- 1 root wheel 1391 13 Nov 2021 /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/usr/include/c++/v1/iostream
+-rw-r--r-- 1 root wheel 1583 13 Apr 2024 /Library/Developer/CommandLineTools/SDKs/MacOSX14.5.sdk/usr/include/c++/v1/iostream
+-rw-r--r-- 1 root wheel 1583 13 Apr 2024 /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/c++/v1/iostream
+-rw-r--r-- 1 root wheel 1583 10 Nov 2024 /Library/Developer/CommandLineTools/SDKs/MacOSX15.2.sdk/usr/include/c++/v1/iostream
+-rw-r--r-- 1 root wheel 1507 8 Mar 03:36 /Library/Developer/CommandLineTools/SDKs/MacOSX15.4.sdk/usr/include/c++/v1/iostream
+-rw-r--r-- 1 root wheel 1507 8 Mar 03:36 /Library/Developer/CommandLineTools/SDKs/MacOSX15.sdk/usr/include/c++/v1/iostream
+I tried on my MacBook, which compiled the test program without issue. However, that was running Monterey, where the workers are running Sequoia. The include paths on my laptop look much better. Where are they configured?
+ +% c++ -v -o test test.cpp
+Apple clang version 15.0.0 (clang-1500.3.9.4)
+Target: x86_64-apple-darwin23.5.0
+Thread model: posix
+InstalledDir: /Library/Developer/CommandLineTools/usr/bin
+ "/Library/Developer/CommandLineTools/usr/bin/clang" -cc1 -triple x86_64-apple-macosx14.0.0 -Wundef-prefix=TARGET_OS_ -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -Werror=implicit-function-declaration -emit-obj -mrelax-all --mrelax-relocations -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name test.cpp -mrelocation-model pic -pic-level 2 -mframe-pointer=all -fno-strict-return -ffp-contract=on -fno-rounding-math -funwind-tables=2 -target-sdk-version=14.4 -fvisibility-inlines-hidden-static-local-var -target-cpu penryn -tune-cpu generic -debugger-tuning=lldb -target-linker-version 1053.12 -v -fcoverage-compilation-dir=/Users/mtelvers/x -resource-dir /Library/Developer/CommandLineTools/usr/lib/clang/15.0.0 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1 -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include -internal-isystem /Library/Developer/CommandLineTools/usr/lib/clang/15.0.0/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -internal-externc-isystem /Library/Developer/CommandLineTools/usr/include -Wno-reorder-init-list -Wno-implicit-int-float-conversion -Wno-c99-designator -Wno-final-dtor-non-final-class -Wno-extra-semi-stmt -Wno-misleading-indentation -Wno-quoted-include-in-framework-header -Wno-implicit-fallthrough -Wno-enum-enum-conversion -Wno-enum-float-conversion -Wno-elaborated-enum-base -Wno-reserved-identifier -Wno-gnu-folding-constant -fdeprecated-macro -fdebug-compilation-dir=/Users/mtelvers/x -ferror-limit 19 -stack-protector 1 -fstack-check -mdarwin-stkchk-strong-link -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fno-cxx-modules -fcxx-exceptions -fexceptions -fmax-type-align=16 -fcommon -fcolor-diagnostics -clang-vendor-feature=+disableNonDependentMemberExprInCurrentInstantiation -fno-odr-hash-protocols -clang-vendor-feature=+enableAggressiveVLAFolding -clang-vendor-feature=+revert09abecef7bbf -clang-vendor-feature=+thisNoAlignAttr -clang-vendor-feature=+thisNoNullAttr -mllvm -disable-aligned-alloc-awareness=1 -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /var/folders/15/4zw4hb9s40b8cmff3z5bdszc0000gp/T/test-71e229.o -x c++ test.cpp
+clang -cc1 version 15.0.0 (clang-1500.3.9.4) default target x86_64-apple-darwin23.5.0
+ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include"
+ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/Library/Frameworks"
+#include "..." search starts here:
+#include <...> search starts here:
+ /usr/local/include
+ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1
+ /Library/Developer/CommandLineTools/usr/lib/clang/15.0.0/include
+ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
+ /Library/Developer/CommandLineTools/usr/include
+ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)
+End of search list.
+ "/Library/Developer/CommandLineTools/usr/bin/ld" -demangle -lto_library /Library/Developer/CommandLineTools/usr/lib/libLTO.dylib -no_deduplicate -dynamic -arch x86_64 -platform_version macos 14.0.0 14.4 -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -o test -L/usr/local/lib /var/folders/15/4zw4hb9s40b8cmff3z5bdszc0000gp/T/test-71e229.o -lc++ -lSystem /Library/Developer/CommandLineTools/usr/lib/clang/15.0.0/lib/darwin/libclang_rt.osx.a
+I’ve been meaning to upgrade my MacBook, and this looked like the perfect excuse. I updated to Sequoia and then updated the Xcode command-line tools. The test compilation worked, the paths looked good, but I had clang 1700.0.13.5, where the workers had 1700.0.13.3.
+ +% c++ -v -o test test.cpp
+Apple clang version 17.0.0 (clang-1700.0.13.5)
+Target: x86_64-apple-darwin24.5.0
+Thread model: posix
+InstalledDir: /Library/Developer/CommandLineTools/usr/bin
+I updated the workers to 1700.0.13.5, which didn’t make any difference. The workers still had that funny /../ path, which wasn’t present anywhere else. I searched /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1 site:stackoverflow.com and the answer is the top match.
++ +Rename or if you’re confident enough, delete /Library/Developer/CommandLineTools/usr/include/c++, then clang++ will automatically search headers under /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1 and find your
+header. That directory is very likely an artifact of OS upgrade and by deleting it clang++ will realise that it should search in the header paths of new SDKs.
I wasn’t confident, so I moved it, sudo mv c++ ~. With that done, the test program builds correctly! Have a read of the answer on Stack Overflow.
Now, rather more cavalierly, I removed the folder on all the i7 and m1 workers:
+ +$ for a in {01..04} ; do ssh m1-worker-$a.macos.ci.dev sudo rm -r /Library/Developer/CommandLineTools/usr/include/c++ ; done
+When you recursively scan a massive directory tree, would you use Sys.readdir or Unix.readdir? My inclination is that Sys.readdir feels more convenient to use, and thus the lower-level Unix.readdir would have the performance edge. Is it significant enough to bother with?
Quickly coding up the two different options for comparison. Here’s the Unix.readdir version, running Unix.opendir then recursively calling Unix.readdir until the End_of_file exception is raised.
let rec traverse_directory_unix path x =
+ let stats = Unix.lstat path in
+ match stats.st_kind with
+ | Unix.S_REG -> x + 1
+ | S_LNK | S_CHR | S_BLK | S_FIFO | S_SOCK -> x
+ | S_DIR ->
+ try
+ let dir_handle = Unix.opendir path in
+ let rec read_entries acc =
+ try
+ match Unix.readdir dir_handle with
+ | "." | ".." -> read_entries acc
+ | entry ->
+ let full_path = Filename.concat path entry in
+ read_entries (traverse_directory_unix full_path acc)
+ with End_of_file ->
+ Unix.closedir dir_handle;
+ acc
+ in
+ read_entries x
+ with _ -> x
+The Sys.readdir version nicely gives us an array so we can idiomatically use Array.fold_left.
let traverse_directory_sys source =
+ let rec process_directory s current_source =
+ let entries = Sys.readdir current_source in
+ Array.fold_left
+ (fun acc entry ->
+ let source = Filename.concat current_source entry in
+ try
+ let stat = Unix.lstat source in
+ match stat.st_kind with
+ | Unix.S_REG -> acc + 1
+ | Unix.S_DIR -> process_directory acc source
+ | S_LNK | S_CHR | S_BLK | S_FIFO | S_SOCK -> acc
+ with Unix.Unix_error _ -> acc)
+ s entries
+ in
+ process_directory 0 source
+The file system may have a big impact, so I tested NTFS, ReFS, and ext4, running each a couple of times to ensure the cache was primed.
+ +Sys.readdir was quicker in my test cases up to 500,000 files. Reaching 750,000 files, Unix.readdir edged ahead. I was surprised by the outcome and wondered whether it was my code rather than the module I used.
Pushing for the result I expected/wanted, I rewrote the function so it more closely mirrors the Sys.readdir version.
let traverse_directory_unix_2 path =
+ let rec process_directory s path =
+ try
+ let dir_handle = Unix.opendir path in
+ let rec read_entries acc =
+ try
+ let entry = Unix.readdir dir_handle in
+ match entry with
+ | "." | ".." -> read_entries acc
+ | entry ->
+ let full_path = Filename.concat path entry in
+ let stats = Unix.lstat full_path in
+ match stats.st_kind with
+ | Unix.S_REG -> read_entries (acc + 1)
+ | S_LNK | S_CHR | S_BLK | S_FIFO | S_SOCK -> read_entries acc
+ | S_DIR -> read_entries (process_directory acc full_path)
+ with End_of_file ->
+ Unix.closedir dir_handle;
+ acc
+ in
+ read_entries s
+ with _ -> s
+ in
+ process_directory 0 path
+This version is indeed faster than Sys.readdir in all cases. However, at 750,000 files the speed up was < 0.5%.
Bella was in touch as the tarides.com website is no longer building. The initial error is that cmarkit was missing, which I assumed was due to an outdated PR which needed to be rebased.
#20 [build 13/15] RUN ./generate-images.sh
+#20 0.259 + dune exec -- src/gen/main.exe file.dune
+#20 2.399 Building ocaml-config.3
+#20 9.486 File "src/gen/dune", line 7, characters 2-9:
+#20 9.486 7 | cmarkit
+#20 9.486 ^^^^^^^
+#20 9.486 Error: Library "cmarkit" not found.
+#20 9.486 -> required by _build/default/src/gen/main.exe
+#20 10.92 + dune build @convert
+#20 18.23 Error: Alias "convert" specified on the command line is empty.
+#20 18.23 It is not defined in . or any of its descendants.
+#20 ERROR: process "/bin/sh -c ./generate-images.sh" did not complete successfully: exit code: 1
+The site recently moved to Dune Package Management, so this was my first opportunity to dig into how that works. Comparing the current build to the last successful build, I can see that cmarkit was installed previously but isn’t now.
#19 [build 12/15] RUN dune pkg lock && dune build @pkg-install
+#19 25.39 Solution for dune.lock:
+...
+#19 25.39 - cmarkit.dev
+...
+Easy fix, I added cmarkit to the .opam file. Oddly, it’s in the .opam file as a pinned depend. However, the build now fails with a new message:
#21 [build 13/15] RUN ./generate-images.sh
+#21 0.173 + dune exec -- src/gen/main.exe file.dune
+#21 2.582 Building ocaml-config.3
+#21 10.78 File "src/gen/grant.ml", line 15, characters 5-24:
+#21 10.78 15 | |> Hilite.Md.transform
+#21 10.78 ^^^^^^^^^^^^^^^^^^^
+#21 10.78 Error: Unbound module "Hilite.Md"
+#21 10.81 File "src/gen/blog.ml", line 142, characters 5-24:
+#21 10.81 142 | |> Hilite.Md.transform
+#21 10.81 ^^^^^^^^^^^^^^^^^^^
+#21 10.81 Error: Unbound module "Hilite.Md"
+#21 10.82 File "src/gen/page.ml", line 52, characters 5-24:
+#21 10.82 52 | |> Hilite.Md.transform
+#21 10.82 ^^^^^^^^^^^^^^^^^^^
+#21 10.82 Error: Unbound module "Hilite.Md"
+#21 10.94 + dune build @convert
+#21 19.46 Error: Alias "convert" specified on the command line is empty.
+#21 19.46 It is not defined in . or any of its descendants.
+#21 ERROR: process "/bin/sh -c ./generate-images.sh" did not complete successfully: exit code: 1
+Checking the hilite package, I saw that there had been a new release last week. The change log lists:
+ +-
+
- Separate markdown package into an optional hilite.markdown package +
Ah, commit aaf60f7 removed the dependency on cmarkit by including the function buffer_add_html_escaped_string in the hilite source.
Pausing for a moment, if I constrain hilite to 0.4.0, does the site build? Yes. Ok, so that’s a valid solution. How hard would it be to switch to 0.5.0?
I hit a weird corner case as I was unable to link against hilite.markdown. I chatted with Patrick, and I recreated my switch, and everything worked.
File "x/dune", line 3, characters 20-35:
+3 | (libraries cmarkit hilite.markdown))
+ ^^^^^^^^^^^^^^^
+Error: Library "hilite.markdown" not found.
+-> required by library "help" in _build/default/x
+-> required by _build/default/x/.help.objs/native/help__X.cmx
+-> required by _build/default/x/help.a
+-> required by alias x/all
+-> required by alias default
+Talking with Jon later about a tangential issue of docs for optional submodules gave me a sudden insight into the corner I’d found myself in. The code base depends on hilite, so after running opam update (to ensure I would get version 0.5.0), I created a new switch opam switch create . --deps-only, and opam installed 0.5.0. When I ran dune build, it reported a missing dependency on cmarkit, so I dutifully added it as a dependency and ran opam install cmarkit. Do you see the problem? hilite only builds the markdown module when cmarkit is installed. If both packages are listed in the opam file when the switch is created, everything works as expected.
The diff turned out to be pretty straightforward.
+ + let html_of_md ~slug body =
+ String.trim body
+ |> Cmarkit.Doc.of_string ~strict:false
+- |> Hilite.Md.transform
++ |> Hilite_markdown.transform
+ |> Cmarkit_html.of_doc ~safe:false
+ |> Soup.parse
+ |> rewrite_links ~slug
+Unfortunately, the build still does not complete successfully. When Dune Package Management builds hilite, it does not build the markdown module even though cmarkit is installed. I wish there was a dune pkg install command!
I tried to split the build by creating a .opam file which contained just ocaml and cmarkit, but this meant running dune pkg lock a second time, and that caused me to run straight into issue #11644.
Perhaps I can patch hilite to make Dune Package Management deal with it as opam does? Jon commented earlier that cmarkit is listed as a with-test dependency. opam would use it if it were present, but perhaps Dune Package Management needs to be explicitly told that it can? I will add cmarkit as an optional dependency.
depends: [
+ "dune" {>= "3.8"}
+ "mdx" {>= "2.4.1" & with-test}
+ "cmarkit" {>= "0.3.0" & with-test}
+ "textmate-language" {>= "0.3.3"}
+ "odoc" {with-doc}
+]
+depopts: [
+ "cmarkit" {>= "0.3.0"}
+]
+With my branch of hilite, the website builds again with Dune Package Management.
I have created a PR#27 to see if Patrick would be happy to update the package.
+ +Feature request for Dune Package Management would be the equivalent of opam option --global archive-mirrors="https://opam.ocaml.org/cache" as a lengthy dune pkg lock may fail due to a single curl failure and need to be restarted from scratch.
I’ve been thinking about terminal-based GUI applications recently and decided to give notty a try.
+ +I decided to write a tool to display the status of the ocurrent/ocluster in the terminal by gathering the statistics from ocluster-admin. I want to have histograms showing each pool’s current utilisation and backlog. The histograms will resize vertically and horizontally as the terminal size changes. And yes, I do love btop.
It’s functional, but still a work in progress. mtelvers/ocluster-monitor
+ +The histogram module uses braille characters (U+2800-U+28FF) to create dense visualizations where each character can represent up to 2x4 data points using the dots of a braille cell. In the code, these positions map to bit values:
+ +Left Column Bits Right Column Bits
+ 0x01 (1) 0x08 (4)
+ 0x02 (2) 0x10 (5)
+ 0x04 (3) 0x20 (6)
+ 0x40 (7) 0x80 (8)
+1. Bit Mapping
+The code defines bit arrays for each column:
+let left_bits = [ 0x40; 0x04; 0x02; 0x01 ] (* Bottom to top *)
+let right_bits = [ 0x80; 0x20; 0x10; 0x08 ] (* Bottom to top *)
+2. Height to Dots Conversion
+let level = int_of_float (height *. 4.0)
+This converts a height value (0.0-1.0) to the number of dots to fill (0-4).
+ +3. Dot Pattern Generation
+For each column, the algorithm:
+-
+
- Iterates through the bit array from bottom to top +
- Sets each bit if the current level is high enough +
- Uses bitwise OR to combine all active dots +
4. Character Assembly
+let braille_char = braille_base lor left_dots lor right_dots
+-
+
braille_base= 0x2800 (base braille character)
+ left_dotsandright_dotsare OR’d together
+ - Result is converted to a Unicode character +
5. Multi-Row Histograms
+For taller displays, the histogram is split into multiple rows:
+-
+
- Each row represents a fraction of the total height +
- Data values are normalized to fit within each row’s range +
- Rows are generated from top to bottom +
Given a list of packages and their dependencies, what order should those packages be installed in?
+ +The above graph gives a simple example of the dependencies of the package dune nicely ordered right to left.
We might choose to model this in OCaml using a map with the package name as the key and a set of the dependent packages:
+ +module PackageSet = Set.Make (String);;
+module PackageMap = Map.Make (String);;
+Thus, the dune example could be defined like this.
let dune = PackageMap.(empty |>
+ add "ocaml" (PackageSet.(empty |> add "ocaml-config" |> add "ocaml-variants")) |>
+ add "ocaml-config" (PackageSet.(empty |> add "ocaml-variants")) |>
+ add "dune" (PackageSet.(empty |> add "ocaml" |> add "base-unix.base" |> add "base-threads.base")) |>
+ add "ocaml-variants" (PackageSet.empty) |>
+ add "base-unix.base" (PackageSet.empty) |>
+ add "base-threads.base" (PackageSet.empty)
+ );;
+We can create a topological sort by first choosing any package with an empty set of dependencies. This package should then be removed from the map of packages and also removed as a dependency from any of the sets. This can be written concisely in OCaml
+ +let rec topological_sort pkgs =
+ match PackageMap.is_empty pkgs with
+ | true -> []
+ | false ->
+ let installable = PackageMap.filter (fun _ deps -> PackageSet.is_empty deps) pkgs in
+ let () = assert (not (PackageMap.is_empty installable)) in
+ let i = PackageMap.choose installable |> fst in
+ let pkgs = PackageMap.remove i pkgs |> PackageMap.map (fun deps -> PackageSet.remove i deps) in
+ i :: topological_sort pkgs
+This gives us the correct installation order:
+ +# topological_sort dune;;
+- : PackageMap.key list =
+["base-threads.base"; "base-unix.base"; "ocaml-variants"; "ocaml-config"; "ocaml"; "dune"]
+I have previously written about using a topological sort of a directed acyclic graph (DAG) of package dependencies to create an ordered list of installation operations. I now want to create a transitive reduction, giving a graph with the same vertices and the fewest number of edges possible.
+ +This is interesting in opam, where a typical package is defined to depend upon both OCaml and Dune. However, Dune depends upon OCaml, so minimally the package only depends upon Dune. For opam, we would typically list both, as they may have version constraints.
+ +depends: [
+ "dune" {>= "3.17"}
+ "ocaml"
+]
+Given a topologically sorted list of packages, we can fold over the list to build a map of the packages and dependencies. As each package is considered in turn, it must either have no dependencies or the dependent package must already be in the map.
+ +let pkg_deps solution =
+ List.fold_left (fun map pkg ->
+ let deps_direct = PackageMap.find pkg solution in
+ let deps_plus_children = PackageSet.fold (fun pkg acc ->
+ PackageSet.union acc (PackageMap.find pkg map)) deps_direct deps_direct in
+ PackageMap.add pkg deps_plus_children map) PackageMap.empty;;
+To generate the transitive reduction, take each set of dependencies for every package in the solution and remove those where the package is a member of the set of all the dependencies of any other directly descendant package.
+ +let reduce dependencies =
+ PackageMap.map (fun u ->
+ PackageSet.filter (fun v ->
+ let others = PackageSet.remove v u in
+ PackageSet.fold (fun o acc ->
+ acc || PackageSet.mem v (PackageMap.find o dependencies)
+ ) others false |> not
+ ) u
+ );;
+Let’s create a quick print function and then test the code:
+ +let print = PackageMap.iter (fun p deps ->
+ print_endline (p ^ ": " ^ (PackageSet.to_list deps |> String.concat ","))
+);;
+The original solution is
+ +# print dune;;
+base-threads.base:
+base-unix.base:
+dune: base-threads.base,base-unix.base,ocaml
+ocaml: ocaml-config,ocaml-variants
+ocaml-config: ocaml-variants
+ocaml-variants:
+- : unit = ()
+And the reduced solution is:
+ +# let dependencies = pkg_deps dune (topological_sort dune);;
+val dependencies : PackageSet.t PackageMap.t = <abstr>
+# print (reduce dependencies dune);;
+base-threads.base:
+base-unix.base:
+dune: base-threads.base,base-unix.base,ocaml
+ocaml: ocaml-config
+ocaml-config: ocaml-variants
+ocaml-variants:
+- : unit = ()
+This doesn’t look like much of a difference, but when applied to a larger graph, for example, 0install.2.18, the reduction is quite dramatic.
+ +Initial graph
+ +
Transitive reduction
+ +
Patrick reported issues with OCaml-CI running tests on ocaml-ppx.
++ +Fedora seems to be having some issues: https://ocaml.ci.dev/github/ocaml-ppx/ppxlib/commit/0d6886f5bcf22287a66511817e969965c888d2b7/variant/fedora-40-5.3_opam-2.3
++sudo: PAM account management error: Authentication service cannot retrieve authentication info +sudo: a password is required +"/usr/bin/env" "bash" "-c" "sudo dnf install -y findutils" failed with exit status 1 +2025-05-12 08:55.09: Job failed: Failed: Build failed +
I took this problem at face value and replied that the issue would be related to Fedora 40, which is EOL. I created PR#1011 for OCaml-CI and deployed it. However, the problem didn’t go away. We were now testing Fedora 42, but jobs were still failing. I created a minimal obuilder job specification:
+ +((from ocaml/opam:fedora-42-ocaml-4.14@sha256:475a852401de7d578efec2afce4384d87b505f5bc610dc56f6bde3b87ebb7664)
+(user (uid 1000) (gid 1000))
+(run (shell "sudo ln -f /usr/bin/opam-2.3 /usr/bin/opam")))
+Submitting the job to the cluster showed it worked on all machines except for bremusa.
$ ocluster-client submit-obuilder --connect mtelvers.cap --pool linux-x86_64 --local-file fedora-42.spec
+Tailing log:
+Building on bremusa.ocamllabs.io
+
+(from ocaml/opam:fedora-42-ocaml-4.14@sha256:475a852401de7d578efec2afce4384d87b505f5bc610dc56f6bde3b87ebb7664)
+2025-05-12 16:55.42 ---> using "aefb7551cd0db7b5ebec7e244d5637aef02ab3f94c732650de7ad183465adaa0" from cache
+
+/: (user (uid 1000) (gid 1000))
+
+/: (run (shell "sudo ln -f /usr/bin/opam-2.3 /usr/bin/opam"))
+sudo: PAM account management error: Authentication service cannot retrieve authentication info
+sudo: a password is required
+"/usr/bin/env" "bash" "-c" "sudo ln -f /usr/bin/opam-2.3 /usr/bin/opam" failed with exit status 1
+Failed: Build failed.
+Changing the image to opam:debian-12-ocaml-4.14 worked, so the issue only affects Fedora images and only on bremusa. I was able to reproduce the issue directly using runc.
# runc run test
+sudo: PAM account management error: Authentication service cannot retrieve authentication info
+sudo: a password is required
+Running ls -l /etc/shadow in the container showed that the permissions on /etc/shadow are 000. If these are changed to 640, then sudo works correctly. Permissions are set 000 for /etc/shadow in some distributions as access is limited to processes with the capability DAC_OVERRIDE.
Having seen a permission issue with runc and libseccomp compatibility before, I went down a rabbit hole investigating that. Ultimately, I compiled runc without libseccomp support, make MAKETAGS="", and this still had the same issue.
All the machines in the linux-x86_64 pool are running Ubuntu 22.04 except for bremusa. I configured a spare machine with Ubuntu 24.04 and tested. The problem appeared on this machine as well.
Is there a change in Ubuntu 24.04?
+ +I temporarily disabled AppArmor by editing /etc/default/grub and added apparmor=0 to GRUB_CMDLINE_LINUX, ran update-grub and rebooted. Disabling AppArmor entirely like this can create security vulnerabilities, so this isn’t recommended, but it did clear the issue.
After enabling AppArmor again, I disabled the configuration for runc by running:
ln -s /etc/apparmor.d/runc /etc/apparmor.d/disable/
+apparmor_parser -R /etc/apparmor.d/runc
+This didn’t help - in fact, this was worse as now runc couldn’t run at all. I restored the configuration and added capability dac_override, but this didn’t help either.
Looking through the profiles with grep shadow -r /etc/apparmor.d, I noticed unix-chkpwd, which could be the source of the issue. I disabled this profile and the issue was resolved.
ln -s /etc/apparmor.d/unix-chkpwd /etc/apparmor.d/disable
+apparmor_parser -R /etc/apparmor.d/unix-chkpwd
+Armed with the answer, it’s pretty easy to find other people with related issues:
+-
+
- https://github.com/docker/build-push-action/issues/1302 +
- https://github.com/moby/moby/issues/48734 +
For some time, we have had issues on Ubuntu Noble when extracting +tar files within Docker containers. See +ocaml/infrastructure#121. +This is only an issue on exotic architectures like RISCV and PPC64LE.
+ +# docker run --rm -it ubuntu:noble
+root@cf3491db4abd:/# cd
+root@cf3491db4abd:~# mkdir foo
+root@cf3491db4abd:~# tar -cf bar.tar foo
+root@cf3491db4abd:~# rmdir foo
+root@cf3491db4abd:~# tar -xf bar.tar
+tar: foo: Cannot change mode to rwxr-xr-x: Operation not permitted
+tar: Exiting with failure status due to previous errors
+The combination of Docker version and libseccomp2 version prevents
+the container from running the fchmodat2 system call. There is a
+bug report on Ubuntu’s bug tracker for the issue.
I have been working around this by building Docker from scratch.
+ +apt install golang
+git clone https://github.com/moby/moby
+cd moby
+AUTO_GOPATH=1 ./hack/make.sh binary
+mv bundles/binary-daemon/* /usr/bin/
+service docker restart
+When provisioning some new RISCV machines, I have once again hit this
+issue, but now the version of Go installed by apt on Ubuntu Noble is
+too old to build Docker!
go: vendor.mod requires go >= 1.23.0 (running go 1.22.2; GOTOOLCHAIN=local)
+As this needs to be repeated multiple times, it makes sense +to wrap the installation steps into an Ansible Playbook. +golang+docker.yml
diff --git a/data/planet/tunbury/user-isolation-on-windows.md b/data/planet/tunbury/user-isolation-on-windows.md new file mode 100644 index 0000000000..ccd401bde8 --- /dev/null +++ b/data/planet/tunbury/user-isolation-on-windows.md @@ -0,0 +1,125 @@ +--- +title: User Isolation on Windows +description: "For a long time, we have struggled to match the performance and functionality + of runc on Windows. Antonin wrote the Docker-based isolation for ocurrent/obuilder + with PR#127, and I wrote machine-level isolation using QEMU PR#195. Sadly, the most + obvious approach of using runhcs doesn\u2019t work, see issue#2156." +url: https://www.tunbury.org/2025/06/09/windows-sandbox/ +date: 2025-06-09T00:00:00-00:00 +preview_image: https://www.tunbury.org/images/sandbox.jpg +authors: +- Mark Elvers +source: +ignore: +--- + +For a long time, we have struggled to match the performance and functionality of runc on Windows. Antonin wrote the Docker-based isolation for ocurrent/obuilder with PR#127, and I wrote machine-level isolation using QEMU PR#195. Sadly, the most obvious approach of using runhcs doesn’t work, see issue#2156.
On macOS, we use user isolation and ZFS mounts. We mount filesystems over /Users/<user> and /usr/local/Homebrew (or /opt/Homebrew on Apple Silicon). Each command is executed with su, then the filesystems are unmounted, and snapshots are taken before repeating the cycle. This approach has limitations, primarily because we can only run one job at a time. Firstly, the Homebrew location is per machine, and secondly, switches are not relocatable, so mounting as /Users/<another user> wouldn’t work.
In a similar vein, we could make user isolation work under Windows. On Windows, opam manages the Cygwin installation in %LOCALAPPDATA%\opam, so it feels like the shared HomeBrew limitation of macOS doesn’t exist, so can we create users with the same home directory? This isn’t as crazy as it sounds because Windows has drive letters, and right back to the earliest Windows networks I can remember (NetWare 3!), it was common practice for all users to have their home directory available as H:\. These days, it’s unfortunate that many applications see through drive letters and convert them to the corresponding UNC paths. Excel is particularly annoying as it does this with linked sheets, preventing administrators from easily migrating to a new file server, thereby invalidating UNC paths.
Windows user isolation
+ +Windows drive mappings are per user and can be created using the command subst. We might try to set the home directory and profile path when we create a user net user foo bar /add /homedir:h:\ /profilepath:h:\, but since h: does not exist in the user’s context, the user is given a temporary profile, which is lost when they log out. If you specify just /homedir, the profile is retained in c:\users\foo.
We could now try to map h: using subst h: c:\cache\layer, but subst drives don’t naturally persist between sessions. Alternatively, we could use net use h: \\DESKTOP-BBBSRML\cache\layer /persistent:yes.
Ultimately, the path where %APPDATA% is held must exist when the profile is loaded; it can’t be created as a result of loading the profile. Note that for a new user, the path doesn’t exist at all, but the parent directory where it will be created does exist. In Active Directory/domain environments, the profile and home paths are on network shares, one directory per user. These exist before the user signs in; all users can have h: mapped to their personal space.
Ultimately, it doesn’t matter whether we can redirect %LOCALAPPDATA% or not, as we can control the location opam uses by setting the environment variable OPAMROOT.
opam knows
+ +Unfortunately, there’s no fooling opam. It sees through both subst and network drives and embeds the path into files like opam\config.
subst
+ +subst h: c:\home\foo
+set OPAMROOT=h:\opam
+opam init -y
+...
+
+ In normal operation, opam only alters files within your opam root
+ (~\AppData\Local\opam by default; currently C:\home\foo\opam).
+
+...
+net use
+ +net share home=c:\home
+net use h: \\DESKTOP-BBBSRML\home\foo /persistent:yes
+SET OPAMROOT=h:\opam
+opam init -y
+...
+
+ In normal operation, opam only alters files within your opam root
+ (~\AppData\Local\opam by default; currently UNC\DESKTOP-BBBSRML\home\foo\opam).
+
+...
+Unless David has some inspiration, I don’t know where to go with this.
+ +Here’s an example from the Windows API.
+ +// If you have: subst X: C:\SomeFolder
+QueryDosDevice(L"X:", buffer, size); // Returns: "C:\SomeFolder"
+GetCurrentDirectory(); // Returns: "X:\" (if current)
+Windows Sandbox
+ +Windows has a new(?) feature called Windows Sandbox that I hadn’t seen before. It allows commands to be executed in a lightweight VM based on an XML definition. For example, a simple test.wsb would contain.
<Configuration>
+ <MappedFolders>
+ <MappedFolder>
+ <HostFolder>C:\home\foo\opam</HostFolder>
+ <SandboxFolder>C:\Users\WDAGUtilityAccount\AppData\Local\opam</SandboxFolder>
+ <ReadOnly>false</ReadOnly>
+ </MappedFolder>
+ </MappedFolders>
+</Configuration>
+The sandbox started quickly and worked well until I tried to run a second instance. The command returns an error stating that only one is allowed. Even doing runas /user:bar "WindowsSandbox.exe test.wsb" fails with the same error.
Full circle
+ +I think this brings us back to Docker. I wrote the QEMU implementation because of Docker’s poor performance on Windows, coupled with the unreliability of OBuilder on Windows. However, I wonder if today’s use case means that it warrants a second look.
+ +# Install Docker Engine
+Invoke-WebRequest -UseBasicParsing "https://download.docker.com/win/static/stable/x86_64/docker-28.2.2.zip" -OutFile docker.zip
+Expand-Archive docker.zip -DestinationPath "C:\Program Files"
+ Environment]::SetEnvironmentVariable("Path", $env:Path + ";C:\Program Files\docker", "Machine")
+
+# Start Docker service
+dockerd --register-service
+Start-Service docker
+Create a simple Dockerfile and build the image using docker build . -t opam.
FROM mcr.microsoft.com/windows/servercore:ltsc2022
+
+# Download opam
+ADD https://github.com/ocaml/opam/releases/download/2.3.0/opam-2.3.0-x86_64-windows.exe C:\\windows\\opam.exe
+
+RUN net user opam /add /passwordreq:no
+
+USER opam
+
+# Run something as the opam user to create c:\\users\\opam
+RUN opam --version
+
+WORKDIR c:\\users\\opam
+
+CMD ["cmd"]
+Test with opam init.
docker run --isolation=process --rm -it -v C:\cache\temp\:c:\Users\opam\AppData\Local\opam opam:latest opam init -y
+Following the setup of rosemary with FreeBSD 14 (with 20C/40T), I have paused spring and summer (which combined have 12C/24T) and rosemary is now handling all of the FreeBSD workload.
+ +Oregano has now taken the OpenBSD workload from bremusa. bremusa has been redeployed in the linux-x86_64 pool. With the extra processing, I have paused the Scaleway workers x86-bm-c1 through x86-bm-c9.
These changes, plus the removal of the Equnix machines, are now reflected in https://infra.ocaml.org.
diff --git a/data/planet/tunbury/zfs-replication-with-ansible.md b/data/planet/tunbury/zfs-replication-with-ansible.md new file mode 100644 index 0000000000..65d397a1ba --- /dev/null +++ b/data/planet/tunbury/zfs-replication-with-ansible.md @@ -0,0 +1,33 @@ +--- +title: ZFS Replication with Ansible +description: "Rather than using the agent-based approach proposed yesterday, it\u2019s + worth considering an Ansible-based solution instead." +url: https://www.tunbury.org/2025/05/16/zfs-replcation-ansible/ +date: 2025-05-16T00:00:00-00:00 +preview_image: https://www.tunbury.org/images/openzfs.png +authors: +- Mark Elvers +source: +ignore: +--- + +Rather than using the agent-based approach proposed yesterday, it’s worth considering an Ansible-based solution instead.
+ +Given a set of YAML files on a one-per-dataset basis containing any metadata we would like for administrative purposes, and with required fields such as those below. We can also override any default snapshot and replication frequencies by adding those parameters to the file.
+ +dataset_path: "tank/dataset-02"
+source_host: "x86-bm-c1.sw.ocaml.org"
+target_host: "x86-bm-c3.sw.ocaml.org”
+The YAML files would be aggregated to create an overall picture of which datasets must be replicated between hosts. Ansible templates would then generate the necessary configuration files for synoid and sanoid, and register the cron jobs on each machine.
Sanoid uses SSH authentication, so the keys must be generated on the source machines, and the public keys must be deployed on the replication targets. Ansible can be used to manage the configuration of the keys.
+ +Given the overall picture, we can automatically generate a markdown document describing the current setup and use Mermaid to include a visual representation.
+ +
I have published a working version of this concept on GitHub. The README.md contains additional information.
+ +The replication set defined in the repository, ZFS Replication Topology, is currently running for testing.
diff --git a/data/planet/tunbury/zfs-send-streams.md b/data/planet/tunbury/zfs-send-streams.md new file mode 100644 index 0000000000..6d5a257f96 --- /dev/null +++ b/data/planet/tunbury/zfs-send-streams.md @@ -0,0 +1,44 @@ +--- +title: ZFS Send Streams +description: We often say that ZFS is an excellent replicated file system, but not + the best local filesystem. This led me to think that if we run zfs send on one machine, + we might want to write that out as a different filesystem. Is that even possible? +url: https://www.tunbury.org/2025/05/02/zfs-send-streams/ +date: 2025-05-02T20:00:00-00:00 +preview_image: https://www.tunbury.org/images/openzfs.png +authors: +- Mark Elvers +source: +ignore: +--- + +We often say that ZFS is an excellent replicated file system, but not the best local filesystem. This led me to think that if we run zfs send on one machine, we might want to write that out as a different filesystem. Is that even possible?
What is in a ZFS stream?
+ +fallocate -l 10G temp.zfs
+zpool create tank `pwd`/temp.zfs
+zfs create tank/home
+cp README.md /tank/home
+zfs snapshot tank/home@send
+zfs send tank/home@send | hexdump
+I spent a little time writing an OCaml application to parse the record structure before realising that there already was a tool to do this: zstreamdump. Using the -d flag shows the contents; you can see your file in the dumped output.
zfs send tank/home@send | zstreamdump -d
+However, this is not like a tar file. It is not a list of file names and their content. It is a list of block changes. ZFS is a tree structure with a snapshot and a volume being tree roots. The leaves of the tree may be unchanged between two snapshots. zfs send operates at the block level below the file system layer.
To emphasise this point, consider a ZVOL formatted as XFS. The structure of the send stream is the same: a record of block changes.
zfs create -V 1G tank/vol
+mkfs.xfs /dev/zvol/tank/vol
+zfs snapshot tank/vol@send
+zfs send tank/vol@send | zstreamdump -d
+ZVOLs are interesting as they give you a snapshot capability on a file system that doesn’t have one. However, some performance metrics I saw posted online showed disappointing results compared with creating a file and using a loopback device. Furthermore, the snapshot would only be in a crash-consistent state as it would be unaware of the underlying snapshot. XFS does have xfsdump and xfsrestore, but they are pretty basic tools.
[1] See also ZfsSend Documentation
diff --git a/data/video-youtube.yml b/data/video-youtube.yml index 6edfa3ffe8..9aa5b33e4f 100644 --- a/data/video-youtube.yml +++ b/data/video-youtube.yml @@ -1,3 +1,27 @@ +- title: Rust for Everyone! + url: https://www.youtube.com/watch/R0dP-QR5wQo?version=3 + thumbnail: https://i3.ytimg.com/vi/R0dP-QR5wQo/hqdefault.jpg + description: 'Rust promises to empower everyone to build reliable software, but + its unique features create steep learning curves. In this talk, Will Crichton + presents four years of research into evidence-based tools that help both novices + and experts work more effectively with Rust''s complexity, including: + + - Ownership visualization: Interactive diagrams showing permissions (read/write/own) + that help students understand borrowing better than rule-based explanations + + - Trait debugging: A graphical tool for navigating complex trait inference trees + when compiler errors become unreadably long + + - Program slicing: Using Rust''s type system to automatically highlight only code + relevant to specific variables + + + All tools discussed are open source and available for use.' + published: 2025-08-27T15:42:33+00:00 + author_name: Jane Street + author_uri: https://www.youtube.com/channel/UCDsVC_ewpcEW_AQcO-H-RDQ + source_link: https://www.youtube.com/feeds/videos.xml?playlist_id=PLCiAikFFaMJoWyXnJ2BWpse5HuiYibNYs + source_title: Jane Street - Tech Talks - title: The Saga of Multicore OCaml url: https://www.youtube.com/watch/XGGSPpk1IB0?version=3 thumbnail: https://i1.ytimg.com/vi/XGGSPpk1IB0/hqdefault.jpg