Dev Docs: New Glossary & JS Search Box

[harding]

This PR adds a new glossary page and a fully Javascript (no server-side scripting) search box that searches keywords for glossary entries, RPCs, BIPs, op codes, and P2P protocol messages.

Developer glossary main page

Preview: http://dg0.dtrt.org/en/developer-glossary

The glossary main page provides a search box and a sorted list of terms. It uses a floating list so that it looks good on both mobiles and desktop/laptop screens.

A link is provided at the bottom of the page to allow readers to recommend new glossary terms. (It works like the Submit New Event link on the Events page.)

Glossary entries

Preview for "block": http://dg0.dtrt.org/en/glossary/block

Each glossary entry has its own page providing a short definition (limited to 255 characters), one or more synonyms, zero or more "not to be confused with" terms, and zero or more links to resources on and off Bitcoin.org. Separate pages were used to allow search engines to link directly to the most relevant entry, rather than forcing users to scroll through a giant page of definitions.

This pull adds:

• 89 glossary entry pages
• 51 additional synonyms
• 222 links (all internal to Bitcoin.org or to Bitcoin Wiki, BitcoinTalk, or Bitcoin StackExchange)

Each page includes links to edit it, view its history, or report an issue about it. Note: those links won't work right until this is merged into master because they're hardcoded for master.

Dev search box

Preview: http://dg0.dtrt.org/en/developer-documentation

A Javascript-powered search box is added to the following pages: Dev Guide, Dev Reference, Dev Examples, Dev glossary, and individual glossary entry pages.

The box allows finding the following by keyword, listed in the order they appear in the search:

• Glossary entries (Bitcoin.org)
• RPCs (Bitcoin.org)
• Op codes (Bitcoin Wiki)
• BIPs (just notable and non-withdrawn BIPs; GitHub.com BIPs repo)
• Bitcoin P2P protocol messages (Bitcoin.org)

The search does use two largish Javascript libraries: JQuery and JQuery UI, both MIT-licensed. I used the plain minified files from upstream for anyone who wants to verify the shasums. This PR adds the libraries to the site, so there's no stats leakage when people load them.

Only the pages that include the search box load the JS libraries and the search data, so other pages on the site are unaffected.

I've tested the search on several Linux desktop browsers and my ancient Android 2.3.5 phone. It works well in all cases.

Backend: autocrossref and links

All glossary pages are built from data files in the _data/glossary/en directory using the new glossary.rb plugin. In addition to rendering the pages, this plugin also generates autocrossreference links and Markdown reference links for the terms.

This was part of the motivation for this pull, as discussed on the mailing list: https://groups.google.com/forum/#!topic/bitcoin-documentation/s_mL1syxYoQ

In the documenation, autoxref links that used to point to other parts of the docs now point to the glossary pages. Readers who want to learn more can follow links from the glossary page back to the docs or on to the other Bitcoin Wiki/BitcoinTalk/Bitcoin StackExchange links.

Also on the backend, this pull passes all previous tests as well as a few new ones.

Commits

I broke this PR into several commits to make review easier. It may be more convenient to review the separate commits rather than the whole diff at once.

[ghost1542]

@harding That is a huge change, thank you so much for your work on this!

I have sent you a pull request with various small fixes and suggestions. I didn't get to check everything in depth, but I have quickly reviewed everything and this LGTM!

I wonder if at some point we could somehow standardize all of this configuration into something more consistent, for instance with a single formatting. As I understand it, data from _data/glossary/en/* currently sends data to _config.yml, _autocrossref.yaml, _includes/references.yml (it's actually used side by side). Although I think it's OK (and jekyll often gives us no choice), the current setup looks a bit intertwined to me. Just in case you'd enjoy spending time on simplifying / standardizing this (if that is even possible).

Again, this is awesome :) Sorry for the time it took for me to review your pull request.

[harding]

@saivann thanks for your review! It was worth the wait :-)

I was also hoping when I started working on this that we could completely get rid of _autocrossref.yml and _includes/references.md, but I couldn't see any reasonably way to put everything from them into the glossary. However, I was thinking that now that we have the glossary entries out of those two files, we can probably merge them into a single file (or part of _config.yml) that basically looks like:

term 1: http://example.com/foo
term 1 alternative name: foo
term 2: /en/developer-documenation

That's possible because we don't need the title text for the links anymore.

And then we'd have a very small plugin that puts them in the format expected by the autocrossref plugin and the Kramdown reference link parser.

However, that sounds like a separate PR to me.

[ghost1542]

@harding Indeed, something like that sounds like an interesting plan! (but sure, preferably at some point in the future to not block this pull request).

[harding]

Added suggested fixes/improvements by @saivann.

In the absence of critical feedback, I'll merge this around 20:00 UTC Friday.

[jlopp]

Should searching for "satoshi" or "denominations" bring up glossary results? It looks like there are definitions for the terms but when I search for them on your test site I don't get any results.

[harding]

@jlopp a few of the common units are auto-cross-referenced to their definitions in the Payment Processing section, as they were before this PR. It would make sense to have a denominations Glossary entry that links to that table as well as to the Wiki's Unit page. I'll make a separate PR for that after this gets merged so it can be properly reviewed. Thanks!

[harding]

Note this was merged and then reverted as we had the same problem with a new plugin as was documented in #773, although because the search box is so prominent, I decided it would be better to revert than to try to fix it live. (At least I was ready for it this time; I wasn't fooled by Travis CI passing.)

I already told @saivann I'll work on fixing the build system so it corresponds with Travis CI, so I'll do that before attempting to merge again.

[ghost1542]

@harding Thanks for keeping a close eye on what happens when merging changes to the plugins. Do you know if it is still caused by inconsistent relative path or something else? The glossary.rb plugin seemed to load files exactly the same way as releases.rb so I didn't expect it would fail.

[harding]

@saivann it was really weird, and I have no idea why it happened. (If it wasn't close to my usual bedtime, I'd look into more.) In case you didn't see, what happened was that if you searched for "locktime" it would direct you to http:///locktime/ and the same for any other term.

I never actually got to a glossary page to see if they built correctly. I should've done that while the reverted tree was building, but I didn't think that far ahead.

I confirmed that my local bulid still works as expected here on top of master, so it's almost certainly another inconsistency between the bundle-based builds and the Ubuntu native Ruby libs build.

[harding]

@saivann I spent several minutes today thinking about the idea I proposed in this comment and I don't think it's as easy or as correct as I initially thought. Unless you have any ideas, I'm going to just leave our link definitions spread over _autocrossref.yaml, _includes/references.md, and _data/glossary/en/* for now and see if something better occurs to me in the future.

@jlopp I'll be starting on the denominations glossary entry after lunch and should have a PR for it up later today. Sorry it took so long to get around to it.

[ghost1542]

@harding Sure!