OCaml.org blog

janestreet — 2023-05-27T08:58:22-00:00

Coming from OCaml, the Rust programming language has many appealing +https://ocaml.org/feed.xmlOCaml.org blog2023-05-27T08:58:22-00:00https://blog.janestreet.com/feed.xmljanestreet

Coming from OCaml, the Rust programming language has many appealing features. Rust’s system for tracking lifetime and ownership allows users to safely express patterns that are awkward in OCaml, such as:

Oxidizing OCaml: Locality

tarides — 2023-05-26T00:00:00-00:00

For the past few years, Tarides has been responsible for the storage component of Tezos, from L1 and L2 shells up to the Tezos protocol. In 2022, our main focus was on improving storage performance and UX for running nodes and bakers. Our efforts resulted in significant improvements, including reducing the storage requirements for rolling nodes by 10x and decreasing the memory usage of the storage layer by 80%. We also maintain core storage APIs necessary to scale the TPS of the network. But we didn't stop there! This year, we've already worked on improving the performance of archive nodes and delivered the project last month. In this blog post, we'll take a closer look at our work and what it means for the future of Tezos.

OCaml Weekly News, 23 May 2023

cwn — 2023-05-23T12:00:00-00:00

OCaml Weekly News, 16 May 2023

cwn — 2023-05-16T12:00:00-00:00

OCaml Weekly News, 09 May 2023

cwn — 2023-05-09T12:00:00-00:00

Optimising Archive Node Storage for Tezos

tarides — 2023-05-05T00:00:00-00:00

Last year we released a garbage collector (GC) for the irmin-pack backend to reclaim disk space. The irmin-pack backend is notably used by Octez to support the Tezos blockchain storage on disk. In this context, the Irmin GC is used to perform context pruning by rolling nodes. This operation reclaims disk space by deleting old blocks that are no longer required to participate in the Tezos consensus algorithm. In Git terms, the rolling nodes only maintain a shallow history of the blockchain. On the other end, archive nodes retain the entirety of the blockchain to ensure integrity. The Irmin GC has the additional benefit of compacting the data on disk, hence improving the performances of disk operations due to better data locality.

Following the context pruning release in Octez v16, we focused on improving the performances of the Irmin GC, both in its disk usage and memory. We also brought the optimisation benefit of data compaction to archive nodes. As the Tezos blockchain keeps growing, the archive nodes storage requirements are also becoming problematic, so our solution opens the possibility to use multiple hard drives to scale beyond a single disk. In the future, this could enable archive node operators to use a small SSD with excellent performance to store the most recent blocks, while storing older blocks on a larger, less expensive disk.

Archive Node Storage Optimisations

@@ -103,7 +103,7 @@ users to safely express patterns that are awkward in OCaml, such as:

Conclusion

The Irmin 3.5 and 3.6 releases brought much needed disk and memory optimisations to Octez context pruning for rolling nodes. The Irmin 3.7 release brings more improvements to rolling nodes and introduces the same optimised garbage collection design to archive nodes, allowing them to have the same small and efficient store for recent history and enabling future improvements for storing history on multiple disks. Integration of Irmin 3.7's features into Octez is in-progress and will ship in a future version of Octez. In the meantime, we welcome your comments and feedback on the optimisations and design choices. Join the conversation on the OCaml Discuss forum, in the GitHub Issues, and through comments on the Tezos Agora post.

Ahrefs is now built with Melange

ahrefs — 2023-05-03T19:06:58-00:00

OCaml, all the way down

Photo by Lance Grandahl on Unsplash

Some history

In 2021, we decided to evaluate Melange as an alternative to ReScript for compiling Ahrefs’ frontend codebase. We wrote about the reasons that led us there, as well as the limitations we encountered at the time, in a previous article.

After this experiment, discussions continued inside the team. Switching to a different compiler, which was in a very early stage, involved quite some risk. But so did the continued use of ReScript, which seemed to be diverging further and further away from OCaml.

Finally, in September 2022 (during ICFP in Ljubljana), we decided to bite the bullet and kicked off a project to deepen the integration between Dune (OCaml’s most used build system) and Melange. This better integration was the key to solve two of the three limitations we had encountered during our initial exploration of Melange:

Build speed would increase due to less work needed to parse dune files, and more efficient rules planning and execution.
Developer ergonomics would get better, as Melange would become a first-class citizen in Dune, with concepts like Dune libraries and other stanzas becoming available to Melange users.

Ahrefs’ leadership backed the project and agreed to financially support the development of this tighter integration. With this support, we set about building a team that included Rudi Grinberg, who maintains Dune as part of its development team, and Antonio Monteiro, who created Melange and is also part of the Dune development team.

Heads down

During the following months, we focused on two tasks, iterating over multiple cycles where the progress on one task would inform the next steps to take for the other:

Evolve Dune to add stanzas, fields, and documentation to support Melange projects.
Migrate Ahrefs’ frontend codebase to use the Melange compiler and Dune, adapt third-party libraries and bindings to Melange, and polish the editor integration, build scripts, and other aspects of the development experience.

We believe that tackling these two tasks in parallel led us to better results, compared to a more waterfall-based approach. As we applied the changes over Ahrefs’ large-ish frontend codebase — it will soon reach 5000 modules — we kept finding and fixing bugs, improving the ergonomics of the Dune and Melange integration, and in general making the solution more robust, real-world ready, and developer-friendly.

Another upside of the way the project was implemented is that we developed it initially in stealth mode, keeping it quite private. By working on it within a tight-knit team, before making a public release, we could make progress faster. We believe that this approach saved future Melange users a lot of churn and burn caused by the multiple changes in Dune stanza options, Melange flags, and other configurations we changed along the way, as we learned more about this integration.

Migration strategies

Initially, our plan was to progressively migrate Ahrefs’ code to Melange. As the frontend codebase is divided into different tools, each being self-contained, we thought we could introduce Melange to build one tool, then another tool, gradually migrating them one by one.

However, this approach turned out to be too complex because configuring a development environment that works on both Melange and ReScript is challenging. As developers could be working on multiple tools during the same week, or even within the same day, we realized that it was unfeasible to reconfigure the environment every time a developer switched from a tool built with Melange to a tool built with ReScript.

Therefore, we changed our minds and opted for a one-shot migration. We would ensure that CI, development, and staging environments were working with Melange and Dune. And we would do this on separate branches, while still using ReScript on our main branch CI and development scripts. Once we were confident everything was building and functioning correctly with Melange, we switched all CI and development scripts to use the Melange and Dune commands. We tried to keep the PR that applied this switch as small as possible, with just a few hundreds of lines of changes so that we could switch back to ReScript if needed. In fact, after a first attempt in March, we had to switch back to ReScript due to some issues on the developer experience side, related to build performance and ergonomics, which took a few more weeks to solve.

In terms of package management and third-party Melange dependencies, we followed a more gradual approach. Dune is quite flexible when it comes to vendoring, so in the initial phase, we downloaded Melange libraries with npm, and had Dune include them in the project as if they were local sources. Now we have started migrating some of these libraries so that we can consume them using opam, the OCaml package manager. This will involve first publishing them in our private opam mirror, but the plan is to have them published in the public opam repository in the future so that other Melange developers can also use them.

Timings

You may be curious about the performance differences between the previous and current approaches. Measuring performance is tricky, but we attempted to measure a few different scenarios with both setups. The results can be seen below.

Keep in mind that Ahrefs frontend setup has specific characteristics, which affect the performance measurements:

Before migration: Dune generated ml files from atd files, then ReScript build tool bsb built all hand-written source files plus the ones generated from atd files.
After migration: everything is built with Dune and Melange.

All measurements were taken on a node with 2x AMD EPYC 7742 cpu @3.2 GHz (nproc=256), 1TB RAM, Debian 11 x86_64 GNU/Linux. The build target is always the entire Ahrefs frontend codebase.

Cold build:

Before: real 0m28.232s, user 9m23.883s, sys 13m33.939s
After: real 1m14.208s, user 10m33.708s, sys 5m45.644s

Warm build, noop (no file is built):

Before: real 0m14.687s, user 3m17.058s, sys 3m57.903s
After: real 0m21.895s, user 0m20.528s, sys 0m1.372s

Watch mode, modifying an “edge” file with almost no reverse dependencies:

Before: 1002ms
After: 1576ms

Watch mode, modifying an “inner” file belonging to a library, with many reverse dependencies:

Before: 7032ms
After: 15394ms

In general, Melange and Dune are slower than ReScript for cold builds in our setup. However, the differences are smaller for warm builds. For watch mode, the difference gets reduced when modifying edge files.

There is room for improvement in the way the Melange and Dune rules are arranged so that cold builds can get faster. For example, delaying some optimizations in Melange might allow to parallelize more work.

Conclusions

The results so far are quite encouraging. These are some of the things that are possible thanks to the deeper integration between Dune and Melange, and its application within the Ahrefs codebase:

The same OCaml compiler is used on both frontend and backend codebases.
Access to all the bug fixes, error improvements, and new features that the OCaml compiler team added between versions 4.06 and 4.14 of the compiler.
A shared developer environment across teams, including editor extensions, OCaml LSP server, etc. No more need to maintain a different set of tooling for backend and frontend.
Removal of hand-written CI checks that were ensuring different tools in the frontend codebase would not access components from other tools. This is now solved by Dune libraries, and the OCaml compiler will complain if logical units try to reach outside their bounds.
Frontend and backend shared dependencies, such as anuragsoni/routes, can now be defined in a single place: an opam file.
Faster rebuilds and better watch mode, as Dune now controls all the build artifacts. Previously, Dune and ReScript were sharing responsibilities, which was leading to unnecessary rebuilds of some artifacts. Or alternatively, rebuilds were not starting when required due to the build system not tracking changes in some subsets of the sources.
Easier PPX maintenance, as there is no longer a need to publish pre-built versions of these tools.
Melange allows to run all ppxs from a single executable file, which has some nice performance benefits.
All the other advantages of using Dune: virtual libraries, watch mode, leverage integrations with tools like odoc…

What’s next?

We are excited about this project becoming a reality, and we believe that the deeper integration between OCaml and Melange through Dune, together with Melange’s ergonomic integration with the JavaScript ecosystem through its bindings, can enable projects that were previously impossible to imagine. For example, full-stack React by hydrating components that are rendered server-side using native OCaml.

Now, we have to make it easier for other people who are willing to try and use Melange and Dune. So our focus is shifting to documenting how Melange works.

There is a section for Melange in the Dune manual that will be included in the next stable release, and that can be consulted today in the latest branch: https://dune.readthedocs.io/en/latest/melange.html.

The next step will be to design and create a site where everyone can read and learn about what is needed to create and maintain a project using Melange and Dune. This site will include a playground, in the spirit of the ReasonML one, so that we can share snippets, see the resulting JavaScript compilation output, and iterate on ideas together.

Besides the above, we have plenty of other things we will be working on in the next months. We will share more information about the roadmap as soon as Dune 3.8 and its respective Melange version are published in the main public opam repository, which should happen in the next weeks.

How to contribute?

If you want to be a part of this, or you want to write or port your libraries to Melange, the best way to do so is by reaching out on the ReasonML Discord. There is a #melange dedicated channel where one can get help and advice on how to get started.

Otherwise, if you are missing features, find bugs, or run into confusing errors, please open an issue in the Melange public repo.

We hope you share our excitement about this update. Our journey to integrate our frontend stack more naturally within the incredible language and ecosystem of OCaml will be well-documented. Stay tuned for further updates in the future!

Ahrefs is now built with Melange was originally published in Ahrefs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Optimising Archive Node Storage for Tezos

tarides — 2023-05-05T00:00:00-00:00

MinidebConf TN 23 was organised by Debian Developers and Villupuram Linux Users Group (VGLUG) as a precursor to DebConf 23 in September at Kochi, India. I had an opportunity to attend and speak at MiniDebConf TN.

Ahrefs is now built with Melange

ahrefs — 2023-05-03T19:06:58-00:00

OCaml, all the way down

Photo by Lance Grandahl on Unsplash

Some history

Build speed would increase due to less work needed to parse dune files, and more efficient rules planning and execution.
Developer ergonomics would get better, as Melange would become a first-class citizen in Dune, with concepts like Dune libraries and other stanzas becoming available to Melange users.

Heads down

During the following months, we focused on two tasks, iterating over multiple cycles where the progress on one task would inform the next steps to take for the other:

Evolve Dune to add stanzas, fields, and documentation to support Melange projects.
Migrate Ahrefs’ frontend codebase to use the Melange compiler and Dune, adapt third-party libraries and bindings to Melange, and polish the editor integration, build scripts, and other aspects of the development experience.

Migration strategies

Timings

Keep in mind that Ahrefs frontend setup has specific characteristics, which affect the performance measurements:

Before migration: Dune generated ml files from atd files, then ReScript build tool bsb built all hand-written source files plus the ones generated from atd files.
After migration: everything is built with Dune and Melange.

All measurements were taken on a node with 2x AMD EPYC 7742 cpu @3.2 GHz (nproc=256), 1TB RAM, Debian 11 x86_64 GNU/Linux. The build target is always the entire Ahrefs frontend codebase.

Cold build:

Before: real 0m28.232s, user 9m23.883s, sys 13m33.939s
After: real 1m14.208s, user 10m33.708s, sys 5m45.644s

Warm build, noop (no file is built):

Before: real 0m14.687s, user 3m17.058s, sys 3m57.903s
After: real 0m21.895s, user 0m20.528s, sys 0m1.372s

Watch mode, modifying an “edge” file with almost no reverse dependencies:

Before: 1002ms
After: 1576ms

Watch mode, modifying an “inner” file belonging to a library, with many reverse dependencies:

Before: 7032ms
After: 15394ms

Conclusions

The results so far are quite encouraging. These are some of the things that are possible thanks to the deeper integration between Dune and Melange, and its application within the Ahrefs codebase:

The same OCaml compiler is used on both frontend and backend codebases.
Access to all the bug fixes, error improvements, and new features that the OCaml compiler team added between versions 4.06 and 4.14 of the compiler.
A shared developer environment across teams, including editor extensions, OCaml LSP server, etc. No more need to maintain a different set of tooling for backend and frontend.
Removal of hand-written CI checks that were ensuring different tools in the frontend codebase would not access components from other tools. This is now solved by Dune libraries, and the OCaml compiler will complain if logical units try to reach outside their bounds.
Frontend and backend shared dependencies, such as anuragsoni/routes, can now be defined in a single place: an opam file.
Faster rebuilds and better watch mode, as Dune now controls all the build artifacts. Previously, Dune and ReScript were sharing responsibilities, which was leading to unnecessary rebuilds of some artifacts. Or alternatively, rebuilds were not starting when required due to the build system not tracking changes in some subsets of the sources.
Easier PPX maintenance, as there is no longer a need to publish pre-built versions of these tools.
Melange allows to run all ppxs from a single executable file, which has some nice performance benefits.
All the other advantages of using Dune: virtual libraries, watch mode, leverage integrations with tools like odoc…

What’s next?

Now, we have to make it easier for other people who are willing to try and use Melange and Dune. So our focus is shifting to documenting how Melange works.

How to contribute?

Otherwise, if you are missing features, find bugs, or run into confusing errors, please open an issue in the Melange public repo.

Ahrefs is now built with Melange was originally published in Ahrefs on Medium, where people are continuing the conversation by highlighting and responding to this story.

OCaml Weekly News, 02 May 2023

cwn — 2023-05-02T12:00:00-00:00

OCaml at MinidebConf TN 2023

tarides — 2023-04-28T00:00:00-00:00

I presented two sessions, one built on our experiences of introducing a Code of Conduct to an open source community (slides here), and one called An Invitation to OCaml, aimed at people with no prior OCaml experience. I was pleased to see a lot of folks getting interested in learning OCaml.

Over the course of two days, I attended interesting sessions by speakers from across India and other parts of the world.

First Day

@@ -118,7 +118,7 @@ users to safely express patterns that are awkward in OCaml, such as:

Renjith, an active Wikipedian, presented their story of moving a Malayalam daily newspaper called Janayugom to an entirely FOSS tech stack. This saved the company a lot of money and stopped Janayugom from shutting down. Renjith emphasised the importance of free speech in a democracy and how small maganizes and newspapers play a role in it. Then Subin presented Varnam, an Indic input tool.

I was impressed by the efforts taken by VGLUG volunteers and the Debian India team to organise everything in a smooth manner. From the time we landed in Villupuram, we did not worry about anything. Transport, food, and lodging were all taken care of by VGLUG volunteers. I did not think such a vibrant community of FOSS users would operate in a rural town. The community is doing great work to uplift the lives of people in their Villupuram.

Best of all, it was great to meet old friends and make new ones. I hope to spread the joy of OCaml in more places.

OCaml at MinidebConf TN 2023

janestreet — 2023-04-28T00:00:00-00:00

Our traders and researchers love Python for its agility and for its huge +

Best of all, it was great to meet old friends and make new ones. I hope to spread the joy of OCaml in more places.

OCaml Weekly News, 25 Apr 2023

cwn — 2023-04-25T12:00:00-00:00

OCaml Weekly News, 18 Apr 2023

cwn — 2023-04-18T12:00:00-00:00

Building reproducible Python environments with XARs

janestreet — 2023-04-14T00:00:00-00:00

Our traders and researchers love Python for its agility and for its huge open-source ecosystem, especially when it comes to machine learning. But the heavy use of notebooks can make it difficult to support. Notebooks have a very different lifecycle than regular code, and aren’t always @@ -126,7 +126,7 @@ rigorously version controlled. And while most of our code (much of it written in OCaml) lives in a monorepo, putting all notebooks there is difficult; many notebooks end up being stored all over the place.

Building reproducible Python environments with XARs

tarides — 2023-04-14T00:00:00-00:00

What’s the best way to spend a Friday evening? We think most people would agree that hacking on OCaml is pretty much at the top of that list (although full disclosure, our sample size for this data could be larger).

OCaml Weekly News, 11 Apr 2023

cwn — 2023-04-11T12:00:00-00:00

OCaml Weekly News, 04 Apr 2023

cwn — 2023-04-04T12:00:00-00:00

OCaml Weekly News, 28 Mar 2023

cwn — 2023-03-28T12:00:00-00:00

Compiler Hacking in Cambridge is Back!

tarides — 2023-03-22T00:00:00-00:00

On Friday the 24th of February, Tarides’s UK office hosted an evening of compiler hacking, presentations, and talks about all things OCaml. We’re continuing a tradition that began in 2013, making this our 19th event, when we (then known as OCaml Labs) were based at the Computer Lab in Cambridge. Just like back then, anyone with an interest in the OCaml compiler is welcome. At our recent event we had a mixture of students, industry professionals, and experts in attendance. If you'd like to create your own compiler hacking sessions, check out the wiki here.

Something that’s changed since 2013 is that OCaml now represents a large chunk of the undergraduate Computer Science tripos at the University of Cambridge; not only as the implementation language for courses such as Compiler Construction & Semantics of Programming Language, but literally as the first language students are taught! This means that we had quite a few undergraduates turn up – it was great to see such an interest in OCaml across different backgrounds.

@@ -186,7 +186,7 @@ difficult; many notebooks end up being stored all over the place.

/static/2a82aadb22ee69e06e9b8c805dd7ce96/c44b8/Patrick.jpg 1360w, /static/2a82aadb22ee69e06e9b8c805dd7ce96/b62ab/Patrick.jpg 3581w" sizes="(max-width: 680px) 100vw, 680px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy" decoding="async"/> -

Talking with Gabriel Nordeborn ReScript, Relay and everything else!

emelle-tv — 2023-03-17T21:20:37-00:00

Compiler Hacking in Cambridge is Back!

emelle-tv — 2023-03-22T00:00:00-00:00

If you would like to sponsor the show contact https://twitter.com/davesnx

OCaml Weekly News, 21 Mar 2023

cwn — 2023-03-21T12:00:00-00:00

Talking with Gabriel Nordeborn ReScript, Relay and everything else!

emelle-tv — 2023-03-17T21:20:37-00:00

Talking with António Monteiro about Melange, Esy, Reason, OCaml and more

emelle-tv — 2023-03-17T21:10:48-00:00

If you would like to sponsor the show contact https://twitter.com/davesnx

Consider sponsoring António's work: https://github.com/sponsors/anmonteiro

cwn

emelle-tv — 2023-05-27T08:58:20-00:00

Craig Ferguson is a software developer at Tarides. Join us in this episode to talk more about OCaml, MirageOS, Irmin and much more!

We are looking for sponsors! If you'd like to support more content for Ocaml, Reason and ReScript just send us a message at emelletvshow@gmail.com -- Watch live at https://www.twitch.tv/emelletv

diff --git a/data/planet-sources.yml b/data/planet-sources.yml index f66fd78049..9bbd850a83 100644 --- a/data/planet-sources.yml +++ b/data/planet-sources.yml @@ -9,3 +9,5 @@ sources: url: https://hannes.robur.coop/atom - name: emelle-tv url: https://watch.ocaml.org/feeds/videos.xml?videoChannelId=23 + - name: cwn + url: https://alan.petitepomme.net/cwn/cwn.rss diff --git a/data/planet/cwn.xml b/data/planet/cwn.xml new file mode 100644 index 0000000000..11aa354144 --- /dev/null +++ b/data/planet/cwn.xml @@ -0,0 +1,2 @@ + +https://alan.petitepomme.net/cwn/cwn.rsscwn2023-05-27T08:58:20-00:00https://alan.petitepomme.net/cwn/cwn.rsscwn

OCaml Weekly News, 16 May 2023

cwn — 2023-05-16T12:00:00-00:00

OCaml Weekly News, 09 May 2023

cwn — 2023-05-09T12:00:00-00:00

OCaml Weekly News, 02 May 2023

cwn — 2023-05-02T12:00:00-00:00

OCaml Weekly News, 25 Apr 2023

cwn — 2023-04-25T12:00:00-00:00

OCaml Weekly News, 18 Apr 2023

cwn — 2023-04-18T12:00:00-00:00

OCaml Weekly News, 11 Apr 2023

cwn — 2023-04-11T12:00:00-00:00

OCaml Weekly News, 04 Apr 2023

cwn — 2023-04-04T12:00:00-00:00

OCaml Weekly News, 28 Mar 2023

cwn — 2023-03-28T12:00:00-00:00

OCaml Weekly News, 21 Mar 2023

cwn — 2023-03-21T12:00:00-00:00