Fix breakage on Kamaelia site

[sparkslabs]

This is somewhat historical at this point, but a number of links on the kamaelia site are bust and the docs generation is currently non-functional. There's a lot of (now somewhat irrelevant) reasons for this, but it would be nice to fix.

One of the best ways of dealing with this is to redo the way the component generation takes place.

In particular -

Get Kamaelia Components to generate their own documentation. This would allow the website docs to be simpler. 95% of the work to do this already exists, but going the final step would probably be a good idea.

Also, this is doable because all the docs in kamaelia are written and intended to be machine readable in the first instance.
The only thing necessary to make that work is to ensure that importing a kamaelia modules does not do any work, and that instantiating a kamaelia object doesn’t do work. But it might be just sufficient to import the module rather than create instances.

Most of the internal docs are currently RST, but it would be nicer if they were markdown. This is akin to asking the classes and modules to generate a __str__ but suitable for external docs, so perhaps add a __markdown__ method (or function) to classes and modules ? That then allows a number of option and having some standard functions for generating aspects of API docs.

[sparkslabs]

Will add comments of things that need fixing. Some initial things:

• MiniAxon just shows the answers next to the questions.
• Formatting is just a bit odd and "tight"
• CSS could be simpler
• Website could be simpler
• On "Index", components are not up to date not updatable at present
• Legacy equivalence between (no .html and html) due to previously using a wiki just confuses the server, browser and relative links - esp in Cookbook
• http://www.kamaelia.org/Publications.html is bust (oddly)
• http://www.kamaelia.org/Presentations.html could be much better done and not use slideshare (which has gone downhill over the years)

Most pages are currently html (as editted by a wiki /client side wiki) rather than a markup, which makes updating a PITA. A build system to rebuild this would help - especially one using markdown instead. Some kind of autoconversation (probably using pandoc) would probably be a good idea.

This is wrong -- http://www.kamaelia.org/Wiki.html (linked on homepage)

http://www.kamaelia.org/ERModeller.html is just plain bust
http://www.kamaelia.org/Whiteboard.html is just plain bust

Batch Transcoder links to http://www.kamaelia.org/ActiveFileProcessor.html which is just broken behaviour

http://www.kamaelia.org/Developers/ is just plain wrong and needs updating to current reality

Many developer pages need reference that the project is currently inactive (even though I'm updating the website) so as not to mislead people.

http://www.kamaelia.org/KamaeliaProjectProcesses.html this especially needs this describing.

This is still a neat idea, but needs improving and more significantly updating with project status -- http://www.kamaelia.org/Developers/Projects

The whole "box right" formatting is just plain whack at the moment :-( -- WIP Fixing where it's used...

http://www.kamaelia.org/About.html -- needs updating to note current project status

Fixed?

• Cookbook links are bust
• Component reference links are bust
• Site is hard to update.
• http://www.kamaelia.org/Components.html is better than the Index, but could still be better
• http://www.kamaelia.org/Repository.html - This is CRAZY out of date
  http://www.kamaelia.org/KamaeliaMacro.html - Has a link to a Adobe Flash object. - Flash died years ago.
  Layout is awful. CSS is pretty dreadful. Images get cropped rather than fit to size (for example)
  Formatting of http://www.kamaelia.org/KamaeliaGrey.html is totally whack
  http://www.kamaelia.org/RecentChanges.html is just completely misleading because a wiki is not being used to edit the website. Needs removing. (Doing this is tricky at present because if the nature of the site) Been removed from some pages manually so far, more to be done and this ideally automated
  Many links in the cookbook are plain wrong. For example this link:
• http://www.kamaelia.org/Cookbook.html.html/Pipelines.htmlAndGraphlines
• Should be this link: http://www.kamaelia.org/Cookbook/PipelinesAndGraphlines This all needs fixing and resolving some on a page by page basis, some en-masse.

[sparkslabs]

Of note, the current rebuild of the docs system currently imports modules and while this requires all the python modules that are used to be installed on the doc build system, this is picking up the fact that a number of the python library modules from other projects that relied upon either no longer exist, don't exist for py3 or simply don't work.

Once things are up and re-running this approach can be re-evaluated again but for the moment this approach allows the docs to be rebuilt at least for PDF, but that's a start for rebuilding the RST/MD/PDF and HTML versions more generally.

[sparkslabs]

(I know I'm "talking" to myself here, these are mainly "notes to self")

[sparkslabs]

Website build process is now updated significantly and now supports build fragments and per page templating. There's a long way to go before the site is up to date and the issues above resolved, but there's a now a devloop, local staging remote staging and publish process in place.

The two Kamaelia "books" that exist - which are the Europython Tutorial (84 pages) and the API reference (>400 pages) are now linked front and centre and "old" downloads are no longer front and centre. I've also added a note about the site being in the process of being updated now that it can be updated sensibly.

A "soon" next step, will be enabling the use of markdown for pages and generating HTML from that to punch into templates - probably using pandoc since from experience that works well as a process for my blog.

A major "plus" at this instant is that the front page no longer contains any "dead" links, which was particularly poor.
Unsurprisingly this is all under local git control. When it's in a saner state, I'll push that repo to github.

[sparkslabs]

Cookbook links are FIXED

[sparkslabs]

Component reference links are FIXED (I think)

[sparkslabs]

Site is hard now not as hard to update.

I have an offline editting system whereby I can:

• Edit files
• Build the site statically offline
• Check it against the old/deployed version
• Serve locally and visually check it
• Sync to current snapshot and upsync to a stage version of the website and confirm things work as expected.
• Deploy that version to "live"

Could still be better but this is a lot better than it was.

[sparkslabs]

Embedded presentations are now working again.

[sparkslabs]

Repository page updated to be not as badly out of date. (Could be improved but this is better)

[sparkslabs]

"Most pages are currently html (as editted by a wiki /client side wiki) rather than a markup"

This is still true, but there's now a pathway to doing this better, with html fragments + metadata first to be followed by markdown-ing.

[sparkslabs]

http://www.kamaelia.org/Wiki.html -- now not linked to from anywhere. Debating what to do with this.

[sparkslabs]

http://www.kamaelia.org/KamaeliaMacro.html - formatting fixed and slideshare embed resolved

[sparkslabs]

"Formatting of http://www.kamaelia.org/KamaeliaGrey.html is totally whack"

It isn't now

[sparkslabs]

Not published/visible yet, but for the purpose of updating and fixing the site, I've resurrected (locally) a version of the wiki software that used to power the website. Using that I've generated a veryplain version of the site. By veryplain it means it uses a template of that name that looks something like this:

    <html>
    <head>
    <title> [[title] ] </title>
    </head>
    <body style="">
    [[content]]
    </body></html>

This lends itself to being put through pandoc to generate markdown and extracting the title - so that we can autogenerate a new version of the website from this version.

So given this I have now also generated a "pure" markdown version of the site. The next step will be to regenerate an HTML version of the site from this pure markdown, but that's the task for another evening.

It is nice to be able to see things like this now though:

AnsweredQuestions.md        GettingStarted.html.md          MiniAxon                          sidebar.md
AUTHOR.md                   GettingStartedInclude.md        MiniAxonFull.html.md              SimpleReliableMulticast.md
Axon-1.1.2-ReleaseNotes.md  GettingStarted.md               MiniAxonFull.md                   Sitemap.md
Axon-1.5.0-ReleaseNotes.md  header.md                       MiniAxon.md                       SomeNewPage.md
AxonShell.md                Home                            MobileReframer.md                 SpeakAndLearn.md
AxonVisualiser.md           index.md                        MulticastRtpMpegRemultiplexer.md  SpeakAndWrite.md
Challenges                  index.orig.md                   navigation.md                     STM.md
c.l.p.a                     Introduction.html.md            NewIntroduction.md                SummerOfCode2006.md
Components                  Introduction.md                 NewPageTemplate.md                SummerOfCode2006Template.md
Components.md               irc.md                          news.md                           SummerOfCode2007.md
Compose                     IRCSpeakerBot.md                News.md                           SummerOfCode2008.md
... [ snip ] ...

When this gets to the stage where this can be used for the website, this will be pushed somewhere public.
For now though this is the first step in getting this to the stage of being practical to edit and update
properly again, so that's a nice set of first steps.

[sparkslabs]

Additionally, I also now have a very trivial script to turn the markdown back into HTML and this is looking very plausible now.

[sparkslabs]

Able to generate html from the markdown for the most awkward page - the Home page - using pandoc, and then style it to look pretty much like the current homepage - which is an important step in auto-generating the site going forward.
It'll also open the door to restyling it much better, since this also becomes almost entirely semantic markup rather than anything else. Quite good for some evening fiddling.

[sparkslabs]

Templated version now exists, and now the styled version of the homepage almost matches the current (legacy) version of the homepage, with a couple of differences - looks slightly better/cleaner and the HTML (generated from markdown largely) is also much simpler and cleaner.

Next step is to autogenerate a new version of the site from the markdown version, then cleanup the markdown, then clean up the autodoc generation.

[sparkslabs]

Have a local build version of the site now that works like this:

• Created a "veryplain" version of the ancient wiki
• This was turned into markdown
• pandoc used to convert that back to plain, simple HTML
• Spike of the homepage styled seperately to make the very basic unstyled HTML look like the current site
• That turned into a template
• Script created to run over all the markdown files
• Structural "boxright" now changed to a semantic "boxright" which is now nicer styled

The result is a pretty clean setup:

• Generator
• Markdown files
• Simple template + CSS
• Static site

This now appears to cover most of the existing public site

Next steps are to check:

• What's missing
• What's out of date
• Anything "bad"
• Broken links
• Fix cookbook

Once this is done, it might be time to switch over to the new site, because it already looks both simpler, but a lot nicer and more consistent.

Title tags also need making consistent/correct.

[sparkslabs]

Revamped site is now in testing.

[sparkslabs]

Revamped website is now live. This means that the breakage on the Kamaelia site is now fixed. The next step will be to update the content. That might take a while but will be follow on issues. There also need to be some local fixes to the build process to match the new system with the old legacy content which can't be managed as markdown, but this is a good starts.

As clarity:

• All html is now generated from markdown files
• All markdown file fragments get stuffed into a simple template
• Much purer semantic CSS now overall.
• Syntax highlighting by default courtesy of pandoc.

\o/

[sparkslabs]

Closing this issue as a result, and will be opening a new parent ticket for content updates.