A web app for visualizing the connection of wikipedia pages. See it at luke.deentaylor.com/wikipedia
Start by entering a topic into the text box, i.e. Cats. A single node will be generated, labeled Cat. Click this node to expand it.
Expanding a node creates a new node for each page that is linked to from the first paragraph of the article. These nodes will be connected to the node from which they were expanded. For example, expanding Cat will create eight nodes, including Fur, Mammal, Carnivore, and Domestication, each of which will be connected to Cat. Each of these nodes can also be expanded in the same way.
You can also enter multiple articles to "compare" by pressing Comma, Tab, or Enter after each one. I've built a custom input to give users visual feedback on this style of input, which behaves similarly to many "tag inputs" found elsewhere.
This project depends on a running Flask server to support the Python API. By default, when you clone this, the
api_endpoint variable (stored in
point to the Flask server on my website. This works fine if you don't want to modify the backend. If you do want to change the backend, you will need to:
- run the Flask server by executing
js/api.jsto point to your running Flask server (typically
I've chosen to use links only from the first paragraph of an article for 2 reasons:
- There is usually a manageable number of these links, about 5-10 per page.
- They tend to be more directly relevant to the article than links further down in the page.
Note that nodes are lighter in color when they are farther away from the central node. If it took 5 steps to reach Ancient Greek from Penguin, it will be a lighter color than a node like Birding, which only took 2 steps to reach. Thus, a node's color indicates how closely an article is related to the central topic.
Hovering the mouse over a node will highlight the path back to the central node: Note that this is not necessarily the shortest path back, but the path that you took to reach the node. Regardless of whether a shorter path back exists, the path by which the node was created will be shown. This is by design.
How it works
When you click to expand a node, a request is made to a Flask server (under the
api directory). This Flask server exposes code from my Python script,
XMLHttpRequests. This Python API parses and json-izes the results of different calls to the Wikipedia API and serves them. The resulting API is at luke.deentaylor.com/wikipedia/api.
The main functionality is getting "first paragraph links", by making a request like http://luke.deentaylor.com/wikipedia/api/links?page=Cats, but this API also exposes methods like getting the name of a random page (which just consists of a simple wrapper around Wikipedia's API)
The underlying script, in
BeautifulSoup to parse through the HTML of wikipedia pages. This HTML is retrieved from calls to Wikipedia's API. The parser looks for the first
<p> tag (non-recursively to exclude informational content inside divs), and excludes certain special edge cases. Then the parser simply finds all links within this paragraph, excluding ones outside of the "Main/Article" namespace. It extracts the page title from each, and returns a list.
The main page
XMLHttpRequest to the aforementioned Flask API. The results of the query are word-wrapped, and then stuck under nodes which are colored according to their distance from the central node, as described above.
Stuff I'd like to implement soon(ish)
- Build a GUI
- Change input method to something other than prompt
- Allow starting anew without refreshing page
- Create small info button that explains the project, controls, etc.
- Render this README into the help dialog
- The area with the network should contain instructions when it is blank
- Create a more thorough help dialog explaining controls, etc. which also includes the README
- Add a "Random Article" button
- Create a better help menu that pops up when a user first visits.
- Make the tour better
- Show users how to expand and trace back nodes. To do this, create a floating invisible div over a start node. Then, pin the Shepherd step to this div.
- Don't allow users to advance to the next step until they've followed the instruction (entering articles, pressing Go)
- Disappear the info box when the tour is started
- Allow inputting of multiple starts
- Build an interface for this
- Implement saving + sharing
- Saving a graph
- Loading a graph from an id
- Loading a graph from a URL parameter
- Implement sharing UI
- Hovering over a node will show a traceback of how you arrived at that node, kind of like breadcrumbs
- mobile optimization: Implement a separate set of controls for touch devices
- On both desktop and mobile, double-click (or tap) a node to open the corresponding wikipedia page in a new tab
- Improve efficiency of highlighting the nodes
.gitignore-ify the libraries directory, no reason for it to be in here when I didn't write that stuff
- Remove dependance on some external libraries:
- Make API requests asynchronous
- Add some kind of build system to make building local copies and contributing easier
Stuff it might be nice to implement sometime in the far future
- Autocomplete names of Wikipedia articles in the top bar
- Make the size of nodes reflect the number of backlinks
- Support for other languages
- Support for other MediaWiki wikis
This project is powered by Wikipedia! Wikipedia is one of my favorite things, and its wealth of information makes this project possible.
Rendering of this document into the help modal is done via a modified version of
marked that supports task lists.