Blog posts in the run up to OSCON

[amirmc]

We've discussed that we will have a series of blog posts in the run up to OSCON so this issue is to track progress of those posts. This issue will be edited as things change/progress and we pick release dates for those posts. Please don't wait for a prompt to get started :)
All posts should be out by 21 July 2014.

Security by @hannesm and @pqwy

• TLS intro (protocol) - 8 July
• nocrypto lib - 9 July
• x509 lib - 10 July
• asn1 lib, parsing and unparsing - 11 July
• ocaml-tls API and internals and attacks/mitigations - 14 July

ocaml-tls and mirage integration (might not do this one)

Others

• ctypes (summary) by @yallop - 15 July
• vchan by @jonludlam - 16 July
• User-level networking by @yomimono - 17 July
• irmin by @samoht - 18 July
• irmin in xen(store) by @djs55 - 21 July
• Minios/ARM by @talex5 - 22 July

conduit by @avsm - 17 July

22 July

• Post on new release (back refs all other posts)
• Post on blog.xen.org for release announcement (back refs all other posts)

[amirmc]

Everyone listed above should already have access to the private OCL papers repo. You can use the 'mirage' folder there for work in progress. Hannes, David, would you mind moving your drafts to that folder?

[hannesm]

@amirmc which mirage folder? I cannot find any... should it be mirage or Mirage?

[amirmc]

I forgot to actually push my changes. Now done.

[hannesm]

done.

[amirmc]

Looking at how much time we have before the announcements. We realistically have 10 days, with one post per day, starting on 7 July (avoiding weekends, if possible). The first week would be the TLS work, as that's closer to being ready, which leaves the following week for the remainder of the posts.

I've just spoken to @yallop and the ctypes post will go live on 14 July, which means we need to figure out when irmin, conduit, vchan and miniOS will go up that week.

[samoht]

I've started a very rough draft for an Irmin manual in https://github.com/samoht/irmin/blob/docs/doc/manual/manual.pdf?raw=true, which I'll try to turn into a blog post in the next days.

[amirmc]

👍

[amirmc]

I've pencilled in provisional dates for the remaining posts. Please can @avsm, @jonludlam and @talex5 confirm that the dates are ok.

[talex5]

The ARM date (draft by 16th) is fine for me.

[avsm]

Confirmed July 15th fine for me.

[hannesm]

for this monday @avsm wants to setup another (or n other) virtual machines with a cloned fs from tls.openmirage, and configure DNS round robin between these machines. the setup currently does use the socket backend of mirage; and cstruct-1.2.0 (thus without the length checking)

[hannesm]

the article for monday https://github.com/ocamllabs/papers/blob/master/mirage/tls-intro.md is ready from our site.. if someone wants to re-read and look over, please do so.

[amirmc]

Thanks! I'll look over tomorrow just to check language and will commit any tweaks directly.

Will comment here when I'm done and will prep a PR for Monday. Are the others going ok? I haven't had a chance to check repo directly.

Best wishes,
Amir

sent via mobile

On 4 Jul 2014, at 17:36, Hannes Mehnert notifications@github.com wrote:

the article for monday https://github.com/ocamllabs/papers/blob/master/mirage/tls-intro.md is ready from our site.. if someone wants to re-read and look over, please do so.

—
Reply to this email directly or view it on GitHub.

[dbuenzli]

A few comments.

• "What is TLS ?" section. I don't really understand the purpose of the second paragraph, suddenly we dive into details of the protocol until a certain point but then the description stops after the handshake. It feels like something was lost in the communication.
• Next section. The name of the section is OCaml-tls, previously the project was mentionend as ocaml-tls, if you start capitalizing I would also capitalize the tls, i.e. OCaml-TLS
• Interop."It renders a sequence diagram of the exchanged messages between your browser and our server by accessing a trace provided by our TLS stack." I find that sentence long-winded.
• Interop. I would end the paragraph with something like "Give us feedback if you encounter any problems".
• I don't know who exactly your target audience is, but the sentence "We implemented effectful front-ends…" will only be meaningful to an OCaml programmer.
• One comment you are likely to get is "they talk about the disadvantage of C blabla but then a good deal of their implementation relies on C primitives" you may want to mitigate that attack in the text. Like "While we rely on cryptographic primitives implemented in C our main reasons for ocaml-tls are that…".

Best,

Daniel

[hannesm]

@dbuenzli thanks! I incorporated your suggestions and extended the article with a section on trusted code base https://github.com/ocamllabs/papers/commit/2e2d3263b131a7e48505dee9b71040bacfb1fec3

[hannesm]

the x509 article https://github.com/ocamllabs/papers/blob/master/mirage/x509.md is also ready for review. I also tested (on a clean system) and adjusted the opam files of the four libraries https://github.com/mirleft/opam-repository

[dbuenzli]

Le samedi, 5 juillet 2014 à 11:51, Hannes Mehnert a écrit :

the x509 article https://github.com/ocamllabs/papers/blob/master/mirage/x509.md is also ready for review.

• First sentence I would remove the , and you want to exchange… and then second sentence instead of the private data, the secure connection.
• s/practise/practice/
• s/thath/that/
• s/mentioned components of the/components mentioned in the/ ?
• server operator and requestor are the same person, I would use the same term to refer to them (there's already enough terminology around).
• "(ignoring recovation…", "(not during a TLS…", remove the parens. In general you may review the whole document and kill a lot of parens, there almost one every two sentence, just integrate that in the sentences.
• 1a but there's no 1b. Is it either 2a or 2b ? In general I didn't understand much out from these bullet points.
• Would be cool to link the code to their definition — warning do link to a specific commit otherwise you may become out of sync in the future as the code base evolves. N.B.
  the syntax identifier is valid.
• Our code, there are many details here, maybe more than I'd like to know, that would however be nice as library documentation. OTOH having it here may entice people to review code, though you should really link all the identifiers.
• To be honest I got a little bit bored, not by your prose, just that it is boring stuff. I doubt most people will read through the whole thing, and this is why I would put the example code before delving into the presentation of your code, as it gives a very good impression of the library API especially compared to that libfetch example. And this the an impression you may want to leave in the 98% readers that will obsessionally switch to another pointless tab.
• At the end of the document there a few missing links (search for [])
• In general I found the presentation a little bit unstructured.

[hannesm]

thanks @dbuenzli https://github.com/ocamllabs/papers/commit/eb95c4116e9e7ba7f652786d3e8d63b84b36f802 -- the reason to link to a specific commit is to not get out of date...

[dbuenzli]

Le dimanche, 6 juillet 2014 à 09:19, Hannes Mehnert a écrit :

the reason to link to a specific commit is to not get out of date...

Yes that's what I was suggesting, I didn't realize you were actually doing that... D

[hannesm]

for tomorrow I'd appreciate to have the libraries in opam. in order to do that, I believe there's a mirage release needed as well (due to the entropy story -- most likely with #262 merged). I'm happy to do the set asn1 / nocrypto / mirage-entropy / tls / x509 PRs to opam-repository tomorrow morning (and update the blog post tls-intro to reflect that the releases are there). any outstanding issues (looking at @avsm and @pqwy) which holds us back from releasing the code?

the tuesday article is not yet in a good shape, maybe we should move asn and x509 to earlier (@pqwy any status of the asn article?)

[pqwy]

The ASN article is ready for a review. I would appreciate any feedback.

@hannesm Should we fail to mold the planned Tuesday one into shape, yes, I think ASN and X509 can go ahead. But let's decide tomorrow.

[dbuenzli]

Le dimanche, 6 juillet 2014 à 22:15, David Kaloper a écrit :

The ASN article (https://github.com/ocamllabs/papers/blob/master/mirage/asn1.md) is ready for a review. I would appreciate any feedback.

• The article is very long (but not a too unpleasant read) so again I don't really expect it to be read by most people. The fact that it is needed to implement TLS appear only far in the text. I would mention in the intro that this is related to ocaml-tls.
• Since it's long it may be worth making an basic, (linked or not), outline as a single sentence in the introduction so that we know what to expect from it. Something like we first provide some background on ASN.1 and then have a closer look at bla bla.
• In a few places there are ( parenthised sentences with spaces around as exemplified ) Usually typographical rules and lispers mandate no spaces between a paren an the word that follows/precedes.
• The polar-src reference is badly formated.
• Rather than parsing combinators I would directly mention pickler combinators which do have the bidirectional property you are after.
  http://research.microsoft.com/pubs/64036/picklercombinators.pdf
• I don't find that the mention to monadic parsers and applicative functors and their problems for your case do bring anything to the paper. It feels like you have to show you know these structure exist. Dropping that would also cut the paper a bit. Also I think that the pickler combinator paper gives you exactly what you eventually need, without having to mention category theory, so it feels pointless to wander around these structures anyway.

[pqwy]

@dbuenzli Thanks, as always 😄

Yes, it's a bit long. I somehow wanted to show the derivation of the library. I'll see if I can cut it down or give more of an outline.

In referencing parsing combinators I am both re-tracing my own decision process, and answering a question that comes up (most recently in the meeting with INRIA people). I grounded this derivation in my work with parser combinators. Other people grounded their questions in their familiarity with parser combinators. No people so far mentioned pickler combinators.

And similarly, I try to frame any parser I even think of designing in terms of Monads or Applicatives so they at least answer a question I would, personaly, immediately have when reading this article. And in reality, although some people claim the names of those signatures had something to do with CT, we all know they were obtained by blind choice from an unusually big dictionary. And now we're stuck with the names.

[pqwy]

The nocrypto article is ready for a review. I would appreciate any feedback.

(Ducks and covers.)

[hannesm]

PRs for opam are in:

• mirage-entropy-unix (requires a mirage-types release) add mirage-entropy-unix package ocaml/opam-repository#2309
• mirage-entropy-xen (requires a mirage-types release) add mirage-entropy-xen package ocaml/opam-repository#2310
• nocrypto Nocrypto release ocaml/opam-repository#2324
• asn1-combinators Asn1 combinators release ocaml/opam-repository#2325
• x509 (on top of nocrypto and asn1-combinators commits so that travis has a chance of building it) X509 release ocaml/opam-repository#2326
• tls (on top of nocrypto, asn1-combinators, and x509 commits; needs a release of mirage {> 1.1.3} with Change mirage tool to apply Entropy_unix functor #262 merged) Tls release ocaml/opam-repository#2327

and now I'm away from my computer for the rest of the day ;)

[dbuenzli]

Le lundi, 7 juillet 2014 à 01:49, David Kaloper a écrit :

In referencing parsing combinators I am both re-tracing my own decision process, and answering a question that comes up (most recently in the meeting with INRIA people). I grounded this derivation in my work with parser combinators. Other people grounded their questions in their familiarity with parser combinators. No people so far mentioned pickler combinators.

Well they may not be up to with the fp litterature. I mean basically you are here only rediscovering pickler combinators, it it were an academic paper (is it research or engineering ?) an I were a reviewer, that's a paper I you would make you cite.

It surely is nice to try to trace the design process, but in some sense I think you'd need to be more precise (tutorial style) in the derivations that end up not to work so that people really get the feel of why it doesn't work. For myself, while reading that I was like, ok so he didn't know about pickler combinators (which is fine) and he shows us how he rediscovered them, cute.

And similarly, I try to frame any parser I even think of designing in terms of Monads or Applicatives so they at least answer a question I would, personaly, immediately have when reading this article.
I wouldn't, for me that's a contrived way of thinking. I tend more to be guided by the structure of the problem, if only to later recognize such structures, rather than try to hammer a screw.

Best,

Daniel

[dbuenzli]

Le lundi, 7 juillet 2014 à 09:35, Daniel Bünzli a écrit :

I wouldn't, for me that's a contrived way of thinking. I tend more to be guided by the structure of the problem, if only to later recognize such structures, rather than try to hammer a screw.

Btw. that's the way cmdliner was designed, it's only after the fact that I realized this was an applicative functor.

Daniel

[dbuenzli]

Le lundi, 7 juillet 2014 à 01:50, David Kaloper a écrit :

The nocrypto article (https://github.com/ocamllabs/papers/blob/master/mirage/nocrypto.md) is ready for a review. I would appreciate any feedback.

• s/behind the TLS project/behind the ocaml-tls project/. Be consistent with branding.
• s/FP principle/functional programming principles/ and I wouldn't link to wikipedia here. Unless you want to give your ADHD diagnosed readers a way out from your blog post.
• "and able to run with no actual operating system" might seem strange for mirage unaware minds, add a justification ?
• "The reason the result converges after one step remained somewhat elusive, though.", what's that in the end ? Is it a bug ? As a reader my curiosity is piqued.
• s/as a primary target of TLS/as a primary target of ocaml-tls/, see for above, for us readers TLS is a protocol, not your project...
• "which does not exist running in a unikernel" is that english ?
• "The entire library looks like a solid, but somewhat outdated piece of engineering." I'm not in a position to judge. You may be right, but I'd be careful with that kind of sentences which could be taken as the kind of comment just made in the air by someone with NI syndrome. I think the points you make before are sufficient to justify the dev. of nocrypto, no need to tease cryptokit's touchy designer further. I would drop that sentence, unless you feel confident enough and want some fun.
• s/with ARC4 implemented/with the vulnerable ARC4/ (not being totally versed in the minutiae of ciphers I had to lookup why you did reserve it a special treatment).
• The sample code is not syntactically valid (toplevel let open), if opening cstruct is only for the to_string I wouldn't open it, either use the qualified name or define an alias.
• s/we are searching/we are looking for/ ?
• "We still haven't implemented this, but certainly intend to". Why not link to an issue that tells us that you are willing to do so ?
• s/simplicity of the TLS library/simplicity of the ocaml-tls library/
• "Above & beyond", why above ? Half of the things you mention are below, new C primitives, GC problems.
• "At this point, it is probably the best cryptographic base library available for our project.", yeah sure you designed it for your project... that feels like a self-congratulating tautology. As a reader I would rather know if it can be useful in my projects too, i.e. if the focus is going to be the minimal needed and designed only with ocaml-tls in mind, or if your mindset is that it should be useful for other projects aswell.

Best,

Daniel

[hannesm]

we revised the article and are happy to get feedback at https://github.com/ocamllabs/papers/blob/master/mirage/tls.md

[dbuenzli]

Le dimanche, 13 juillet 2014 à 21:56, Hannes Mehnert a écrit :

we revised the article and are happy to get feedback at https://github.com/ocamllabs/papers/blob/master/mirage/tls.md

Comments on the article

• We have been forking on for the past six months, github joke ?
• OCaml ecosystem -> The OCaml ecosystem
• "There were also two distinct basic "platforms" we wanted to target from the outset: the case of a simple executable, and the case of unikernels. We wanted our library to support both of those, too." These two sentence say exactly twice the same thing.
• to it fullest extent -> to its fullest extent
• new session values and resulting bytes -> maybe new session values and new resulting bytes to make it really clear no byte is being mutated.
• the errors are signaled through the result type -> the errors are signaled through the ret type.
• XXX make this break more natural XXX: yes. You just presented the handle_tls signature above so what comes after this line would be better just following that signature. The signature of "Other entry points" need to go in Core API where they are discussed. The paragraph following the XXX and their entry points are interesting but somehow a little bit loosely connected.
• processes each individually -> process each individually
• One question that came to me was whether the Core function calls are sufficiently fast to be cooperative (which makes it much easier to make it independent from concurrency libraries as you don't need to cps to account for yielding, e.g. as I do in uutf, jsonm etc., that is you don't need to return an awaiting state).
• Core API, linkify the various API entry points to their mli.
• stateful objects -> stateful structures (objects have a particular meaning in OCaml).
• pull-based, I'd call that push, it always depend on which side you look at things (the client pushes data, callbacks pull data) so maybe it's better not to use that confusing terminology. The main question for me is whether it is blocking or non-blocking. Initial hand-shake: do you mean it's a blocking call ? I think these two paragraph are a little bit confusing, I'm pretty sure you are doing the right things, i.e. non-blocking all the way along to be compatible with cooperative concurrency but somehow these two paragraph make me doubt.
• plans for plain blocking unix, why not plain non-blocking unix, that way you could trivially support concurrency libraries on top of it (see e.g. https://github.com/dbuenzli/nbcodec/, and uutf or jsonm as concrete examples) with overall less code surface.
• Attack on TLS, kill some parens there.
• collision of MD5] -> collision of MD5
• The original introduction indicated in black bold "Please be aware that this release is a beta and is missing external code audits. It is not yet intended for use in any security critical applications.", this document enjoins me to "and run our code for their services". That seems contradictory.

Overall an interesting read. It would certainly nice to keep (and update !) the attacks part with the code base.

Comments on snippets of code I saw:

• design of type ret: tagged tuples can be represented by records in OCaml…
• Coding style. I personally do not like empty white lines in function bodies. They are useless, lead to uselessly uncompact code (e.g. initial lets in the the client function) and especially make it much harder to scan the code/delimitate the extent of functions as we don't do braces in OCaml. Also more whitespace doesn't mean more readable. I see no reason to put a space in front of a ';' (especially since it's wrong and ugly from a typographical point view). Also trying to align the -> in pattern matches makes it harder for me to find the end of the patterns, and scanning/delimiting them and their bodies becomes harder aswell. See the ocaml programming guidelines. http://caml.inria.fr/resources/doc/guides/guidelines.en.html
• client_exn/server_exn, is generating an invalid configuration a programming error ? In that case I'd rather raise Invalid_argument and remove that _exn unless you are fan of Hungarian notation. If it's not then the functions should return a variant rather than raise.

Best,

Daniel

[pqwy]

Thanks @dbuenzli! I made a few changes to reflect your suggestions and I suspect @hannesm will make even more in the morning.

Believe it or not, "forking" was an honest typo.

[yallop]

Draft ctypes post here (rendered). Comments very welcome; pull requests even more so.

[samoht]

Irmin post is ready to review (rendered). Comments and patches are very welcome.

[amirmc]

After brief discussion, we decided to swap the order of the last two posts. This give us some more time to try out the ARM work.

[avsm]

@talex5 @amirmc can we rearrange the ARM post to Tuesday to keep the two Irmin posts together?

That will also give us a few days to figure out some last-minute blockers on getting mirage-www compiling with ARM.

[amirmc]

We're supposed to have the release announcement post out on Tues. That post must go up as the link to it is already in the news release.

Best wishes,
Amir

sent via mobile

On 19 Jul 2014, at 15:11, Anil Madhavapeddy notifications@github.com wrote:

@talex5 @amirmc can we rearrange the ARM post to Tuesday to keep the two Irmin posts together?

That will also give us a few days to figure out some last-minute blockers on getting mirage-www compiling with ARM.

—
Reply to this email directly or view it on GitHub.

[amirmc]

All the posts linked in the news release are at mirage/mirage-www#168 (comment)

The only two left to go up are the ARM post and 2.0 announcement.

[avsm]

That's fine, we can do two posts then I guess.

On 19 Jul 2014, at 15:31, Amir Chaudhry notifications@github.com wrote:

We're supposed to have the release announcement post out on Tues. That post must go up as the link to it is already in the news release.

Best wishes,
Amir

sent via mobile

On 19 Jul 2014, at 15:11, Anil Madhavapeddy notifications@github.com wrote:

@talex5 @amirmc can we rearrange the ARM post to Tuesday to keep the two Irmin posts together?

That will also give us a few days to figure out some last-minute blockers on getting mirage-www compiling with ARM.

—
Reply to this email directly or view it on GitHub.
—
Reply to this email directly or view it on GitHub.

[amirmc]

Yup, will have to be if we want another Irmin post before Tues.

I'd say two posts on Tues but they both have to be out before embargo lifts (will double-check what time that is).

Best wishes,
Amir

sent via mobile

On 19 Jul 2014, at 15:43, Anil Madhavapeddy notifications@github.com wrote:

That's fine, we can do two posts then I guess.

On 19 Jul 2014, at 15:31, Amir Chaudhry notifications@github.com wrote:

We're supposed to have the release announcement post out on Tues. That post must go up as the link to it is already in the news release.

Best wishes,
Amir

sent via mobile

On 19 Jul 2014, at 15:11, Anil Madhavapeddy notifications@github.com wrote:

@talex5 @amirmc can we rearrange the ARM post to Tuesday to keep the two Irmin posts together?

That will also give us a few days to figure out some last-minute blockers on getting mirage-www compiling with ARM.

—
Reply to this email directly or view it on GitHub.
—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub.

[avsm]

recap post live at http://openmirage.org/blog/announcing-mirage-20-release

[amirmc]

companion post also live on the Xen Blog at http://blog.xen.org/index.php/2014/07/22/mirage-os-v2-0-the-new-features/

[amirmc]

We did it! Thank you everyone for the huge effort over the last couple of weeks. All posts are live and we've had a lot of visibility in the meantime, which is fantastic.

For reference, you can check out the official press release [1], which links back to the blog posts and is now being picked up by news sites. That means even more people will be finding their way to the content over the coming days and weeks.

Great work!

[1] http://www.marketwired.com/press-release/xen-project-introduces-new-mirage-os-release-1931602.htm

PS you have no idea how much I'm looking forward to clicking the close button. :)

[samoht]

Congrats, that was nicely coordinated!

[avsm]

Three cheers for Amir, and all the blog contributors! Now, who's up for a second round of posts?? sound of leaves drifting quietly through a forest

On 22 Jul 2014, at 09:25, Thomas Gazagnaire notifications@github.com wrote:

Congrats, that was nicely coordinated!

—
Reply to this email directly or view it on GitHub.

[samoht]

@amirmc your blog post doesn't seem to appear on the front page of http://www.xenproject.org/

[amirmc]

Well spotted. I've pinged Russ about it so hopefully will be fixed soon.

Best wishes,
Amir

sent via mobile

On 22 Jul 2014, at 17:40, Thomas Gazagnaire notifications@github.com wrote:

@amirmc your blog post doesn't seem to appear on the front page of http://www.xenproject.org/

—
Reply to this email directly or view it on GitHub.