[WIP] Conf refactoring portland 2020

[mxsasha]

This contains very much in progress work in a refactored conference website for Portland 2020, starting from Prague 2019.
It contains a lot of data from 2019 applied to 2020 for now, so things like schedule rendering can be tested based on fully filled content. Styling is perhaps also off.

Todo list for schedule processing changes:

• Fix links to speakers
• Check whether "convince your manager" talks rendering still works
• Refactor new Python code and look at performance
• Possible spurious warnings from core.py when flaghasschedule is False

Some other things I'm thinking of:

• Fix the enormous yearly duplication of static assets
• Remove unused static files
• Review other things than schedule for duplication / whether they can be more data driven
• Consider flattening some legacy conf websites to plain html
• See if we can reduce the yearly conference CSS size a bit (10K lines seems excessive)
• Set correct info for Portland 2020
• Clean up commits
• Jot down some useful directions for review

Possibly move to future PR:

• Have validation/schema for all yaml files
• Review referenced tickets
• See whether we can remove duplication of menu templates
• See if docs/include and docs/_templates/include should be merged/reorganised
• Documentation for conf websites
• Look at splitting off some of the sponsorship data into includes (packages are per conference)
• Consider including 2013 and 2014 websites, currently on github pages

[mxsasha]

This PR is (eventually) intended to address at least #681, #674 and #480.

[plaindocs]

I'm guessing we'll just abandon or restart #749 from scratch when this goes in?

[plaindocs]

Hey @mxsasha could you possible look at normalizing the includes directories? We currently have

• docs/include/
• docs/_templates/include

with the theory I think that the first should include rst and the second html/jinja but I'm pretty sure we're not at all consistent. And I've a nagging feeling that there is a third include dir somewhere I've forgotten about.

[mxsasha]

@plaindocs @ericholscher Prompted by looking into the includes directories, I did some research on the option of archiving conference websites. We talked about this a few times, but it is a significant decision, so we should take a moment to think about it.

It would mean that we can move legacy conf websites out of the main docs folders. Their CSS and static image assets would remain as they are, but all their RST, MD, base templates, YAML schedule and includes would be removed. The YAML that contains the speaker info remains, because that is also a source for video info right now. I would want to do this for all conferences from 2017 and older. We can repeat this process every year or so.

The process for this would be to render them to static HTML, then place that in html_extra_path, so the Sphinx build process will just copy the HTML over. It's quite easy to set up, now that I know how.

Upsides:

• Reduces legacy cruft in our repo, allowing easier edits and refactoring
• Increases build speed

Downsides:

• If we ever want to use Sphinx to render other formats than HTML, e.g. PDF or plaintext, these sites can not be included. However, we're already quite tied to HTML output, so I think this does not matter.
• Making edits in the old sites will be harder.

I am leaning towards doing this. Any thoughts?

[mxsasha]

Also, regarding #492 I am looking at integrating mobile and desktop menus in the site. The structure is too different to have a single template for mobile and desktop, with the content. It would work if I extract the nav data entirely to YAML. As in, have the entire navigation tree in the conference yaml config file. The buttons are alread there. I’m inclined to think that YAML-based nav would be progress, but maybe there is still something I’m overlooking?

[mxsasha]

I have looked into reducing the duplication of conference template files (venue.rst, job-fair.rst, etc) and concluded this is not worth it with Sphinx.

Sphinx HTML is very much built around the "one input file, one output file" model. There are some places we could hook into with a custom builder, but even then there are other problems - we can't just write out the same HTML file multiple times, we actually have to render it multiple times with different contexts. Changing this would mean deeply hooking into several parts of Sphinx. This brings risks of greatly increased complexity, we don't want to rewrite half of Sphinx. The benefit is minor, because we're only talking about a handful of files per conference. Perhaps worth it if we have 20 conferences per year, but not for our current amount.

[ericholscher]

I am leaning towards doing this. Any thoughts?

I've looked at this in the past, and I hit random blockers. I'm not 100% sure what they were, but if it isn't super complicated I'm +1 on it.

The buttons are alread there. I’m inclined to think that YAML-based nav would be progress, but maybe there is still something I’m overlooking?

+1. Definitely moving it into YAML and out of HTML is a win (similar to sponsors)

I have looked into reducing the duplication of conference template files (venue.rst, job-fair.rst, etc) and concluded this is not worth it with Sphinx.

Likely the easiest way to do this is via symlinks, but that doesn't really solve the problem. I think having a file that does an include is probably fine, and lets us modify it for a specific year or event, which we often have to do. Eg. for the list of projects on Writing Day, or the list of companies in the job fair.

[mxsasha]

Because this branch was not rebased onto master, but a merge was done, I can't do any further changes to commits, so those can't be cleaned up any further. Due to the size of some of the commits, we should definitely use a non-squash option to merge this PR, to retain individual commits.

For review, the individual commits are fairly descriptive, but the best way to see how things work may be to look at the new state: https://github.com/writethedocs/www/tree/conf-refactoring-portland-2020

I'll summarise the most significant changes:

Naming of YAML files in _data is now consistent: {shortcode}-{year}-[config/schedule/sessions]. Sessions is the file formerly named speakers, schedule is a new format. With alphabetic sorting, this keeps related files closer.

The conference config is largely unchanged, except a new setting total_talk_days. The templatecode hack has been removed.

The new schedule format is much simplified, and is now a single schedule for all days. It uses total_talk_days to figure out how many days to expect. A slot can either have a title, which can be an arbitrary text, or a slug, which refers to a session in the sessions YAML file. Warnings are emitted for slugs that can not be found in the sessions file.

The code loading the conference info has been significantly extended. It is entirely backwards compatible. For 2020 and newer conferences, it expects the new format. As a side effect, this now means that all conference templates have access to the full conference schedule with all metadata about all sessions, in their rendering context. This only applies if the conference flags are set, we don't look for a schedule unless flaghasschedule is set, and same for flaghasspeakers. As loading this data is a lot more complicated, conference data is now cached during the Sphinx build, for old and new conferences.

Static files for conferences have been dramatically cleaned up. This also means we no longer need to copy all images for every year, they have one image directory (they were almost entirely identical). Per-year CSS variations are now handled by creating a new CSS file per year rather than an entire static files directory.
Many unused static files were also removed from the whole static directory, and a lot of different folders for images were merged together so that each file has a fairly clear place.

Conference websites <= 2017 are now static HTML. This means that all their includes and base templates are deleted, which cleans up a lot. The static files are still present, although many have been moved away to a legacy folder, as the overlap is limited with 2018 and newer conference sites. Data files were also removed except their speaker info, as that's used for videos. The static HTML is automatically copied into the build directory by sphinx.

[plaindocs]

@mxsasha that sounds incredible. <3

[ericholscher]

I couldn't get gulp to run locally -- what node version are you using? I tried with v12.9.1 and v8.11.2.

[plaindocs]

Naming of YAML files in _data is now consistent: {shortcode}-{year}-[config/schedule/sessions]. Sessions is the file formerly named speakers, schedule is a new format. With alphabetic sorting, this keeps related files closer.

Should we bother also doing the config and the sessions under the same scheme?

The new schedule format is much simplified, and is now a single schedule for all days. It uses total_talk_days to figure out how many days to expect. A slot can either have a title, which can be an arbitrary text, or a slug, which refers to a session in the sessions YAML file. Warnings are emitted for slugs that can not be found in the sessions file.

Super cool! Can we make that a breaking error?

The code loading the conference info has been significantly extended. It is entirely backwards compatible. For 2020 and newer conferences, it expects the new format. As a side effect, this now means that all conference templates have access to the full conference schedule with all metadata about all sessions, in their rendering context. This only applies if the conference flags are set, we don't look for a schedule unless flaghasschedule is set, and same for flaghasspeakers. As loading this data is a lot more complicated, conference data is now cached during the Sphinx build, for old and new conferences.

<3

We should add a flagsoldout as per recent Prague tickets

Conference websites <= 2017 are now static HTML. This means that all their includes and base templates are deleted, which cleans up a lot. The static files are still present, although many have been moved away to a legacy folder, as the overlap is limited with 2018 and newer conference sites. Data files were also removed except their speaker info, as that's used for videos. The static HTML is automatically copied into the build directory by sphinx.

Do we need to document 'how to move confs to legacy?'

[plaindocs]

Might make sense to conditionalize in the HTML based on flagislandingpage ?

[plaindocs]

I've poked around here and there, and can't find anything broken, I'd be inclined to merge it in.

[mxsasha]

Naming of YAML files in _data is now consistent: {shortcode}-{year}-[config/schedule/sessions]. Sessions is the file formerly named speakers, schedule is a new format. With alphabetic sorting, this keeps related files closer.

Should we bother also doing the config and the sessions under the same scheme?

They are, for 2020 and newer.

Super cool! Can we make that a breaking error?

I'll have another look, but I had some difficulty with that before.

We should add a flagsoldout as per recent Prague tickets

Yes.

Do we need to document 'how to move confs to legacy?'

Yes, that'll go into the follow-up PR.

Overall, I strongly suggest merging this if it works for us, and not adding any additional work or bugfixes, except for bugs introduced in this PR (i.e. not including #1065). The remaining work can go into a new (less huge) PR.

[mxsasha]

I couldn't get gulp to run locally -- what node version are you using? I tried with v12.9.1 and v8.11.2.

It is now also broken for me, running gulp style. I think somewhere in the merging with master, the dependencies or something got broken. This is really the part I know the least about.

[mxsasha]

Broken regarding gulp meaning:

/Users/sasha/dev/writethedocs-www/gulpfile.js:14
gulp.task('styles', function(callback_finished) {
    ^

SyntaxError: Unexpected token .
    at Module._compile (internal/modules/cjs/loader.js:760:23)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:827:10)
    at Module.load (internal/modules/cjs/loader.js:685:32)
    at Function.Module._load (internal/modules/cjs/loader.js:620:12)
    at Module.require (internal/modules/cjs/loader.js:723:19)
    at require (internal/modules/cjs/helpers.js:14:16)
    at execute (/Users/sasha/dev/writethedocs-www/node_modules/gulp-cli/lib/versioned/^4.0.0/index.js:36:18)
    at Liftoff.handleArguments (/Users/sasha/dev/writethedocs-www/node_modules/gulp-cli/index.js:201:24)
    at Liftoff.execute (/Users/sasha/dev/writethedocs-www/node_modules/liftoff/index.js:201:12)
    at module.exports (/Users/sasha/dev/writethedocs-www/node_modules/flagged-respawn/index.js:51:3)

[ericholscher]

I fixed this conflict and merged it from the CLI 👍

[plaindocs]

🎉 awesome work folks!