QA: Site Build Problem Follow-Up

[harding]

This morning, #bitcoin IRC user waxwing reported that all of the tables in the RPC and REST section were broken:

Recalling a similar situation from about a year ago, I was able to quickly push an apparent fix in commit 84f5b6f. This issue isn't about the bug itself but about preventing future problems.

I find this issue particularly concerning because I can't replicate it locally, and the same commit that produced the broken page also compiles cleanly on Travis. I suspect that the build server is compiling the site differently from the way I compile it, and that's the source of the immediate problem, so my first recommendation is:

Select one canonical way to compile the site and make sure that both Travis and the build server use it. Also describe it in the README.

The Makefile has a test that would've caught this problem if the site was built through make rather than by calling bundle exec jekyll build directly. One concern we've had with the Makefile is that it may implement tests that require programs/libraries not installed on the build server. To deal with that, my next recomendation comes in two parts:

1. I push a commit adding a site-build target to the Makefile that builds the site and only runs tests that we know work on the build server. Travis and users can still use make all to run all tests.
2. We build the site using make site-build

Finally, I'd like to know if the live site differs from my local site, so I suggest we add to the Makefile a post-build step that runs a command like this:

find _site -type f | xargs sha256sum > _site/sha256sums.txt

And then another Makefile target that can be manually run to diff the local _site/sha256sums.txt against the bitcoin.org/sha256sums.txt, showing whether any files built locally are different from the current site files. (This is not a security step, just a QA step.)

Some files will likely always differ, such as the events page that includes data from Meetup or the two pages that include commit counts from GitHub---but I think most of the rest of the pages are reasonably deterministically built and should be identical between local and remote builds. In short form, I recommend:

I create a pull request adding the above-described commands to the Makefile.

Also, for discussion, I think the site building script should be part of the repository. Since anyone who has commit access can push a commit adding an rsh server to the _plugins directory, it seems like executing the build from a file in the repository doesn't introduce any new attack surface. However, I'm happy to discuss this further and also review the current build script for problems before making it public.

(Note: as discussed previously on another issue, the site is not built on the same server as the one that hosts the files, so commit access does not give anyone access to the Bitcoin.org binaries.)

[saivann]

@harding Good catch!

I still don't understand why the relative path was inconsistent in this specific case and not elsewhere (looks like jekyll's fault)... I now feel sorry for not enabling Makefile tests on the build server, I used them for testing locally but applying them to the build machine was still somewhere on my todo list. I have just fixed and tested the build script so this kind of issue should be catched in the future. The build server uses a conventional linux distribution so there should be no issues with installing additional dependencies.

Regarding the idea of open-sourcing the build script, I am fine with it. Maybe other server scripts should be open-sourced at the same time (translation previews, stats, torrent). I'd also like to suggest I give you SSH access to the build machine. You'd not have access to binaries and webserver, but you'd be able to debug and fix build issues with the website if I am not around.

Regarding the sha256sums idea, I feel you'd likely have a very hard time with the events and contributors pages, as well as potential small unimportant differences in the outputs of different jekyll versions.

[harding]

@saivann thanks, and thanks to waxwing for reporting it!

I have just fixed and tested the build script so this kind of issue should be catched in the future.

Thanks!

Regarding the idea of open-sourcing the build script, I am fine with it.

Cool!

I'd also like to suggest I give you SSH access to the build machine [...] to debug and fix build issues with the website if I am not around.

I'd appreciate that, and I'm glad its isolated.

Regarding the sha256sums idea, I feel you'd likely have a very hard time with the events and contributors pages,

Not worried about that, as we can simply either manually ignore those or automatically whitelist them by filtering the results with grep.

as well as potential small unimportant differences in the outputs of different jekyll versions.

I don't think that should be a problem since you, me, Travis, and the build server are all using bundler and the Gemfile. In any case, I'm willing to spend a couple hours trying it implement it. It's easy to run Makefile tests and ad hoc tests against files locally, but all that testing is fruitless if we can't be sure that the actual site has the same content.

[harding]

Closing as all proposed steps have been implemented and make manual-checks now works. (It even let me discover a small site build discrepency now fixed in 2a2df42)