[3211] Convert to build process

[rossjrw]

SCP-3211 has been historically built using a spreadsheet. I would like to convert it to a more sensible build process.

• Convert janky spreadsheet concatenation to a template file
• Convert HTML template to Markdown
• Create a foolproof collapsible component thanks html5!
• Make the final collapsible have a margin-bottom only when closed
• Create a foolproof footnote component
• Create a foolproof image block component
• Make an en dir and be reasonably sure that there are no English strings in the main dir that will end up in the output (warning and loading sections included!)
• Convert all the sub-anomalies to config files
• Create a build process that outputs a single ftml file (and any dependencies) per language
• Optimise on space - diffing?
• Get styling working
• Get functionality working
• Add instructions for development and translating
• Add a note about file generation to the top of the output, with a link to contribution instructions
• Implement Gulp or something, so the functionality is passed through Babel but the build stuff isn't (and rewrite the build stuff as Babel, possibly)
• Add all the images (with the SVG in en/)
• Recover image quality (cwebp defaults to 75; vectorlike images (cube, puce, imgmissing) should be 100)
• Port across any changes that were made to the wiki but not to the spreadsheet
• Consider porting across translations too (CN + KO + RU) (I might have to apply to their site in order to update the page? Maybe I should set up an auto-updater first) Considered! I will do this another time in separate PRs
• Change the concept of the 'review' section to just the anomaly section but displaying the base anomaly
• Generate the iframe JS using Webpack
• Add some sort of rinse step that leaves only dist files in dist (or maybe move build files into build/)
• Tell Kaktus he gets the esteemed title of Longest SCP By Page Source Character Count back
• Copy any files that are intended to be uploaded into each language's dist dir
• Grep for XXX and TODO; resolve

[rossjrw]

Bearing in mind the previous length was 197519, at this point the total length of all sources is 171397, which if I implement no optimisations, gives me a little under 30000 characters to play with. The current functionality and styling is 12837 characters. That makes 184234, excluding fiddly bullshit like indentation and credits. The hard limit is 200000. I'd need to beat 199659 to claim longest article and 198583 to claim longest fiction. I retain longest SCP so long as I stay above 175372.

I'd really like to implement these optimisations, or at least try to, so I suspect I'll be seceding the title to SCP-1730. The next longest SCP (SCP-4231) is sitting at 120904 so I'll pretty comfortably take 2nd place. Obviously taking the top spot with SCP-3211 is wildly dishonest - as far as I know the other longest articles are free of duplicated content.

Update: the diffing optimisation saves me about 3000 characters per iteration. Compare that to the previous optimisation, which was about 1500 characters per iteration.

I'm a bit disappointed, but it's better than nothing.

Actually, it seems that the diff contains the replaced strings from the reference material in every diff, so theoretically, the size of each diff could be cut down by half. It might be much more efficient to use a 100% generic reference document instead of the base anomaly. Problem with that, though, is that there's a lot of logic in the document dedicated to simplifying the configuration for the base anomaly where the lack of such simplifications would effect huge amounts of effort needed to make the rest of the anomalies consistent with what the base anomaly would need to be - for example, every time Breach says "it is nothing" in even a slightly different way, a new string would need to be added to all of the anomalies.

Given that the purpose of this refactoring is to make rewriting it easier, I decide now that it is definitely not worth the effort.

Might it be as easy as using the ejs template as the reference? UPDATE: Yes - I am now saving about 8000 characters per iteration. The total saving is 68% and I've even added some output stats on the command line.

[rossjrw]

Interfacing between Gulp and Webpack is not as intuitive as I'd hoped.

build.ts outputs an array of deltas, which are needed (and declared) in iframe.ts. Declaring implies those values are already present rather than e.g. passed to a wrapper function, and I'm cool with that assumption. It would probably be fine to have those deltas be globally declared in a script element in the final iframe HTML.

It's hard to visualise which things are dependent on languages. Is bundle.js language-dependent, for example? (Answer: yes, I think so? warning.svg is definitely language-dependent, at least, and that will be in the bundle. I think it would probably just be easier to create dist/en/, dist/cn/, etc with no common files between them. If that means regenerating the bundle a bunch of times, so be it.

You know, I'm actually starting to think that maybe Webpack is not the tool to use for this. There's just too many moving parts, too many things that are dependent on each other. Let's try just Gulp.

[rossjrw]

Build is working fine so far using just Gulp, but I've reached a predictable block: the iframe JS isn't bundled. I need to resolve the imports and stuff for the browser. Webpack is definitely the best tool to use for this, but luckily, I'm now working from a more sensible mindset. It's no longer an all or nothing situation. This time, I'll be using Webpack for JS only.

Webpack is rolling, everything is pretty much working as much as I'd expect at this point, but the bundle is a lot bigger than I'd like, and the major culprit is jsondiffpatch, which seems to be pretty much entirely space-unoptimised. I think I'd be able to save a lot of space by rolling my own solution, especially given that my use case is so simplified (because my patches are only additive).

Alternatively - I could unpkg/jsdelivr the dependencies. This would include not only jsondiffpatch but potentially js-cookie and compress-tag too (I've been using unpkg's distribution of js-cookie since the start).

Hmmm. I think I'm going to switch to unpkg for now and see what happens, with the caveat that if I end up using this permanently and unpkg ever goes down, I can easily switch to pure Webpack. Actually, no - I'm going to unpkg js-cookie, remove compress-tag, and roll my own patcher.

[rossjrw]

Deployed. Everything seems fine.

...except the double-hyphen-to-em-dash converter also converts the double hyphen in the wdfiles URL. So now I've got to find some way of getting around that that's not a blanket find-and-replace.