Test the code examples

[tombentley]

It's embarrassing when the website contains example code which doesn't compile. We need a way to check that all the valid examples actually compile, and all the supposedly invalid ones really do not compile.

This shouldn't be so hard to do. Walking the directories to find .md files is easy. Any line indented at least 4 spaces (and not preceeded with a <!-- lang: ... --> annotation) is Ceylon code. Any code block with a line matching ^.*// *ERROR.*$ should be invalid. The remaining problem is examples which are bare statements, or expressions. We can't easily detect those and annotating them would be tedious, but perhaps we can simply try compiling code and if it fails, try putting it in a dummy method. That might not be perfect, but it would probably be good enough.

[chochos]

This could be fully automated... it's "just" a matter of sending the detected code via HTTP POST to try.ceylon-lang.org and parsing the response JSON to see if it was valid or not (meaning did it compile or not). It would be extremely useful given that all those snippets will be eventually runnable using that same service.

[quintesse]

Remember that we should by trying to link as much of those code samples to http://try.ceylon-lang.org so people can see them work and play around with them

[tombentley]

@chochos I think it would be better done operating on the markup, as we can use <!-- --> annotations (like we do for specifying the language used for an example) to encode metadata about the snippet. Looking at some of the example I'm writing sometimes exmaple 2 depends on a declaration in example 1, for instance. A simple <!-- depends: previous --> could indicate that, but once it's rendered to HTML you've lost that info.

Similarly some bare expressions really require more than just a dumb wrapper method as I thought above. I've just written a snippet like this:

String attr = this.attr;

Which requires a whole class and method for it to make sense. Putting that in the example itself would destroy the readability of the thing I'm trying to describe. Maybe for stuff like that a annotation would be needed, or better a way of specifying the surrounding boilerplate, but in a not-too-obtrusive way.

@quintesse Generating the markup for links to http://try.ceylon-lang.org would be best done using the awestruct pipeline. I'll raise another issue for that.

[chochos]

I was actually talking about extracting the snippets from the markup, not the rendered html.

[emmanuelbernard]

That would definitively be a life and time saver down the line. Though we need to be able to differentiate versions of ceylon the code is supposed to represent. That's for the future that is.

[chochos]

I'll get to work on this after M2 is released

[tombentley]

I've hacked something together which should suffice for now and will use it find/fix problems for M2.

[tombentley]

Where should we put the site checker tool that I've been using? It doesn't belong in ceylon-compiler, but nor does it belong in ceylon-lang.org either.

[emmanuelbernard]

@tombentley What is your site checker made of? Can you push it in a peronal repo on https://github.com/tombentley so we can more easily discuss it?

[tombentley]

I wrote it in Java, see https://github.com/tombentley/cow

[emmanuelbernard]

Ok that's rather specific to Markdown and how we display things in the website. But putting it in ceylon-lang.org repo might make things difficult.
I tend to think a separate repo is best. Not sure.

Why cow btw?

[tombentley]

Well, the middle bits are specific to markdown, but there's nothing to stop you fitting in different way of extracting code blocks and annotations, or from validating non-Ceylon code. Though I'm sure Gavin would object if its nice spec document grew lots of annotation warts ;-)

Why cow? No good reason.