Rebuild GDC site as a vibe.d server, converting all pages to markdown. #6

ibuclaw · 2014-06-22T19:43:30Z

Rather than using diet templates, it functions in the following way:

Receives a request object for GET /foobar
Translates /foobar into a path of where the markdown file is expected to be.

'/'         => /views/index.md
'/foo'      => /views/foo.md
'/foo/bar/' => /views/foo/bar.md

Checks if the translated path exists as a file and proceeds to render the page in the topdown order of:
Template header (templates/header.inc)
Markdown script (views/foobar.md)
Template footer (templates/footer.inc)
Sends the built page back to the response object.

The benefit of this being that templates can be loaded onto the server dynamically and without requiring a restart of the entire service to make a change.
As of writing, the memory consumption and build time was > 50% reduction in comparison to using Diet templates to compile only four pages in CTFE.

The only time when a site rebuild would be required is for fixing any of the todo listed items in src/app.d.

Before:

After:

ibuclaw · 2014-06-22T21:12:36Z

@jpf91 - also, lets talk implementing /downloads into this server.

mihails-strasuns · 2014-06-22T22:55:00Z

src/app.d

+
+  // Catch all GET requests and push them through our main handler.
+  auto router = new URLRouter;
+  router.get("*", toDelegate(&handleRequest));


If you only a single request handler, router is not necessary, you can do straight listenHTTP(settings, &handleRequest); (in fact router is implemented as multiplexing handler)

Actually, if I do it that way, wouldn't the handler need to check the type of request? (As in, discards anything that isn't a GET request)

Yes. Though I don't see why you would want to reject any other HTTP requests other than trying to go for RESTful style semantic purity.

Actually, I tested various commands, all came back as bad requests, so I guess your suggestion is fine.

Yes, RESTful style semantic purity. eg: GET and POST should not perform the same action. ;-)

jpf91 · 2014-06-23T15:33:42Z

@ibuclaw I think the simplest approach for downloads/index.html is using a downloads/index.json file containing the data and downloads/index.mustache containing the template. As mustache has got a D implementation, we can use it instead of compile time diet templates. Header/footer can be handled like all other pages. I guess the mustache template can generate markdown+html and we can then pass it through filterMarkdown. If you agree with this concept I can provide the code later today.

Also moving part of our email discussion here:
I actually only wanted to keep two or three releases on the download page. This way users can immediately access the previous release in case there are regressions in GDC or the frontend or if they haven't updated their code yet. All older releases can then be browsed on the FTP server.
So do you think we should provide more history and split up the page?

OT caching is probably important for the download page. @Dicebot does vibe-d have some nice caching helpers? I guess the simplest approach is to stat the template files and only re-render if the timestamp changed. However, this still produces some disk-accesses for every request. I guess for huge sites a 'fast path' which uses a time check could be useful.

string cachedSite()
{
    static string cachedString;
    static SysTime lastCheck;
    //Hopefully compiles down to a simple integer add / comparison
    if(Clock.currTime() < lastCheck + dur!msecs(100))
        return cachedString;
    else
    {
        if(sourceFileChanged())
        {
            cachedString = renderPage();
        }
        lastCheck = Clock.currTime();
        return cachedString;
    }
}

mihails-strasuns · 2014-06-23T15:40:50Z

@jpf91 not something I am aware of. But it can be implemented by built-in tools of reverse proxy in front of vibe.d, for example proxy_cache in case of nginx.

ibuclaw · 2014-06-23T16:31:26Z

@jpf91 I was thinking some sort of output range / custom AA to store cached pages.

Notable fields:

struct CacheStore
{
   immutable hash_t key;  // requestURL path
   hash_t checksum;       // mod-time & size of path
   string content;        // GC alloc'd data
}

As for downloads, I think things should be split up/separated in a nicer way. That goes perhaps for the ftp view too. If it's going to be used as an archive of releases, then the toplevel should be a directory named after the version number.

downloads/
. gdc-4.6/
. gdc-4.8/
.. 2.064.2/
.... arm-linux-eabi/
.... i686-linux-gnu/
.... x86_64-linux-gnu/
.. 2.065.0/
.... arm-linux-eabi/
.... i686-linux-gnu/
.... x86_64-linux-gnu/
. gdc-4.9/
. gdc-latest/     <-- gdc-4.9
. gdc-oldstable/  <-- gdc-4.8

But then again, maybe the downloads page can be cleaned up using only CSS changes.

jpf91 · 2014-06-23T17:00:14Z

Do you think we need an archive html page for 'old releases'? Otherwise we could simply keep the most recent release on downloads.html and restructure the ftp view as you proposed.

jpf91 · 2014-06-23T17:29:10Z

ibuclaw#1

ibuclaw · 2014-06-23T19:37:19Z

I don't see a need for old releases/archives. For the D front-end, it should always be the latest (2.065). For GCC, it should be OK to keep 2 versions (4.9.x, 4.8.x) as there are those who are still on older systems. :)

ibuclaw · 2014-06-28T18:51:22Z

And (more as a joke) with the GNU head.

AlixPexton · 2014-06-29T09:44:32Z

I detect clip-art in those wilder designs! The whole wolf thing was a joke anyway, one I'm starting to regret ><

I don't know if you noticed, but one of the doodle sheets I shared had a C that looked like a Gnu on its side, kinda/sorta/ifyasquintabit. I was thinking of putting it on bottom face of an inside-out cube, but it looked awful.

I also doodled a G that is made out of an n and a u stacked on top of each other, but I didn't think anyone would be interested in that.

AlixPexton · 2014-06-29T09:51:19Z

I just took a look at a few reference images for inspiration, and it turns out that wolves have a dark area around their eyes that is gibbous moon shaped. Turns out I already drew some wolves without noticing ^^

ibuclaw · 2014-06-29T10:30:08Z

I detect clip-art in those wilder designs! The whole wolf thing was a joke anyway, one I'm starting to regret ><

It's all drawn by my own hand-ish in inkscape. I say hand-ish because I'm limited to a trackpad on my laptop. I'm also not much of an artist behold simple stuff. So clip-art would be a nice way of describing my work. :)

ibuclaw · 2014-07-28T06:33:52Z

I've actually been working on a visual overhaul since this pull: http://staging.dgnu.org

So I will go ahead and merge this (just need to do a cursory rebase to make sure everything is in order) and raise a new pull for the new-new site.

@jpf91 - Any thoughts on what I've changed on the downloads page?

Rebuild GDC site as a vibe.d server, converting all pages to markdown.

jpf91 · 2014-07-28T17:09:09Z

The new design looks great!
But on the download page I'd either reduce the spacing between the 'host' headers and the download boxes or increase the spacing between the release archive and the 'host' header.

BTW: The download tables don't scale to well for very small windows (same problem with the old design) but I couldn't figure out how to make that work.

ibuclaw · 2014-07-30T22:02:17Z

Done! I've also right aligned the link too (as per the old design).

Yeah, I need to work out how to best layout the tables.

ibuclaw · 2014-07-31T06:41:05Z

OK, I've tweaked the downloads page, it's not as fancy as before, but it is more consistent now on smaller screens vs larger.

jpf91 · 2014-07-31T12:40:24Z

That's much better :-) One last nitpick: can we keep alternating colors for the table rows?

ibuclaw · 2014-07-31T13:03:20Z

They are alternated. Albeit there is not much difference between the two shades of grey. :)

jpf91 · 2014-07-31T13:48:34Z

Weird, I could swear I used ctrl-f5 to reload the page... Anyway with caches cleared I now get these alternating colors, so I think this is good to go.

ibuclaw · 2014-07-31T14:31:20Z

Anyway with caches cleared I now get these alternating colors, so I think this is good to go.

Files are checked for changes every 5 seconds. So time for cache to refresh is....

jpf91 · 2014-07-31T16:20:56Z

@ibuclaw looks like the download links don't work for some reason?

ibuclaw · 2014-07-31T16:56:20Z

I noticed. Somehow it tries to retrieve (on the server)
downloads/downloads/binaries/i686-w64-mingw32/dgreen/gcc-4.6.1-tdm64-1-gdc-6a4bafe80b-20120708-D2.058.7z

MartinNowak · 2014-11-08T00:53:31Z

What was the reason you decided against diet?
While diet is compiled it'd still be possible that updating the downloads.json generates a new page at runtime.

ibuclaw · 2014-11-08T17:57:37Z

From what I recall of testing, when you make a change to a diet template, you must recompile (taking down the server with it) in order to load in the new changes.

However, I can't test now as it seems that vibe.d no longer builds with D2.065

MartinNowak · 2014-11-09T09:09:34Z

From what I recall of testing, when you make a change to a diet template, you must recompile (taking down the server with it) in order to load in the new changes.

Correct and interesting to hear that this was the deal breaker. I usually compile my sites on my development machine, upload the binaries and restart the systemd service. So there is no real downtime.

ibuclaw · 2014-11-09T11:04:11Z

Correct and interesting to hear that this was the deal breaker. I usually compile my sites on my development machine, upload the binaries and restart the systemd service. So there is no real downtime.

Except for the small blip when you restart the service. ;-)

One of the nice things about this was that I can add an unlimited number of pages without the need for recompiling. I just push the changes in and they are there on the server.

ibuclaw · 2014-11-09T11:08:37Z

However, I can't test now as it seems that vibe.d no longer builds with D2.065

Actually, I was attempting to build with my experimental copy of 2.066, hmm...

mihails-strasuns reviewed Jun 22, 2014
View reviewed changes

ibuclaw mentioned this pull request Jun 28, 2014

New Logo Design #7

Merged

ibuclaw and others added 3 commits July 28, 2014 10:38

Rebuild GDC site as a vibe.d server, converting all pages to markdown.

813a29c

Add code to generate downloads/index.html

8c4b524

Update against old site for markdown/vibe.d conversion.

328b786

ibuclaw added a commit that referenced this pull request Jul 28, 2014

Merge pull request #6 from ibuclaw/markdown

3b699a9

Rebuild GDC site as a vibe.d server, converting all pages to markdown.

ibuclaw merged commit 3b699a9 into D-Programming-GDC:master Jul 28, 2014

ibuclaw deleted the markdown branch July 31, 2014 14:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rebuild GDC site as a vibe.d server, converting all pages to markdown. #6

Rebuild GDC site as a vibe.d server, converting all pages to markdown. #6

ibuclaw commented Jun 22, 2014

ibuclaw commented Jun 22, 2014

mihails-strasuns Jun 22, 2014

ibuclaw Jun 23, 2014

ibuclaw Jun 23, 2014

mihails-strasuns Jun 23, 2014

ibuclaw Jun 23, 2014

jpf91 commented Jun 23, 2014

mihails-strasuns commented Jun 23, 2014

ibuclaw commented Jun 23, 2014

jpf91 commented Jun 23, 2014

jpf91 commented Jun 23, 2014

ibuclaw commented Jun 23, 2014

ibuclaw commented Jun 28, 2014

AlixPexton commented Jun 29, 2014

AlixPexton commented Jun 29, 2014

ibuclaw commented Jun 29, 2014

ibuclaw commented Jul 28, 2014

jpf91 commented Jul 28, 2014

ibuclaw commented Jul 30, 2014

ibuclaw commented Jul 31, 2014

jpf91 commented Jul 31, 2014

ibuclaw commented Jul 31, 2014

jpf91 commented Jul 31, 2014

ibuclaw commented Jul 31, 2014

jpf91 commented Jul 31, 2014

ibuclaw commented Jul 31, 2014

MartinNowak commented Nov 8, 2014

ibuclaw commented Nov 8, 2014

MartinNowak commented Nov 9, 2014

ibuclaw commented Nov 9, 2014

ibuclaw commented Nov 9, 2014

Rebuild GDC site as a vibe.d server, converting all pages to markdown. #6

Rebuild GDC site as a vibe.d server, converting all pages to markdown. #6

Conversation

ibuclaw commented Jun 22, 2014

ibuclaw commented Jun 22, 2014

mihails-strasuns Jun 22, 2014

Choose a reason for hiding this comment

ibuclaw Jun 23, 2014

Choose a reason for hiding this comment

ibuclaw Jun 23, 2014

Choose a reason for hiding this comment

mihails-strasuns Jun 23, 2014

Choose a reason for hiding this comment

ibuclaw Jun 23, 2014

Choose a reason for hiding this comment

jpf91 commented Jun 23, 2014

mihails-strasuns commented Jun 23, 2014

ibuclaw commented Jun 23, 2014

jpf91 commented Jun 23, 2014

jpf91 commented Jun 23, 2014

ibuclaw commented Jun 23, 2014

ibuclaw commented Jun 28, 2014

AlixPexton commented Jun 29, 2014

AlixPexton commented Jun 29, 2014

ibuclaw commented Jun 29, 2014

ibuclaw commented Jul 28, 2014

jpf91 commented Jul 28, 2014

ibuclaw commented Jul 30, 2014

ibuclaw commented Jul 31, 2014

jpf91 commented Jul 31, 2014

ibuclaw commented Jul 31, 2014

jpf91 commented Jul 31, 2014

ibuclaw commented Jul 31, 2014

jpf91 commented Jul 31, 2014

ibuclaw commented Jul 31, 2014

MartinNowak commented Nov 8, 2014

ibuclaw commented Nov 8, 2014

MartinNowak commented Nov 9, 2014

ibuclaw commented Nov 9, 2014

ibuclaw commented Nov 9, 2014