Pageviews Analysis tool for Wikimedia Foundation wikis
Switch branches/tags
2018.09.24T03.03 2018.08.29T14.54 2018.08.09T18.38 2018.08.08T16.40 2018.07.26T00.48 2018.07.17T14.20 2018.06.19T15.32 2018.06.18T16.29 2018.05.31T14.40 2018.04.24T21.03 2018.04.16T01.44 2018.03.18T19.31 2018.03.13T00.28 2018.01.24T18.32 2018.01.24T00.17 2018.01.14T03.00 2017.11.06T06.40 2017.10.27T16.29 2017.09.16T22.06 2017.07.26T03.28 2017.07.01T02.46 2017.05.23T21.02 2017.05.04T17.07 2017.04.05T17.08 2017.03.12T03.59 2017.03.07T16.57 2017.02.24T18.17 2017.02.21T18.24 2017.02.18T00.47 2017.02.15T02.36 2017.02.13T23.37 2017.02.09T03.13 2017.02.08T00.47 2017.02.04T22.06 2017.02.02T00.00 2017.01.29T06.14 2017.01.28T21.57 2017.01.16T04.19 2017.01.09T21.45 2017.01.09T00.56 2017.01.02T05.17 2016.12.27T07.21 2016.12.18T05.39 2016.12.06T04.41 2016.12.05T02.17 2016.12.03T01.00 2016.11.26T19.17 2016.11.17T19.21 2016.11.17T06.18 2016.11.15T00.300 2016.11.12T19.07 2016.11.10T01.16 2016.10.25T16.23 2016.10.18T03.07 2016.09.29T00.32 2016.09.28T20.07 2016.09.23T04.30 2016.09.19T15.48 2016.09.11T22.20 2016.09.08T16.56 2016.08.24T05.08 2016.08.22T00.07 2016.08.10T03.24 2016.08.08T21.10 2016.08.06T20.56 2016.08.05T07.16 2016.07.28T22.25 2016.07.22T12.39 2016.07.21T01.05 2016.07.20T22.23 2016.07.18T23.19 2016.07.17T02.58 2016.07.14T20.25 2016.07.10T14.04 2016.07.08T21.08 2016.07.05T16.11 2016.07.04T20.15 2016.06.30T23.30 2016.06.30T02.21 2016.06.29T18.54 2016.06.23T15.55 2016.06.21T23.17 2016.06.07T16.21 2016.06.05T18.42 2016.06.04T17.56 2016.06.03T21.57 2016.05.19T19.21 2016.05.19T15.45 2016.05.17T11.40 2016.05.16T20.41 2016.05.15T21.46 2016.05.15T19.25 2016.05.15T18.19 2016.05.14T16.01 2016.05.13T15.40 2016.05.12T16.38 2016.05.12T05.11 2016.05.11T18.40 2016.05.10T17.57 2016.05.10T03.37
Nothing to show
Clone or download
Permalink
Failed to load latest commit information.
javascripts Add missing wikis, wikimediafoundation.org -> foundation.wikimedia.org Sep 25, 2018
jsdocs Pageviews Analysis 2.0!!!!!!! Nov 12, 2016
messages Localisation updates from https://translatewiki.net. Oct 15, 2018
public_html Add missing wikis, wikimediafoundation.org -> foundation.wikimedia.org Sep 25, 2018
stylesheets Collapse nav links under More menu if they don't fit Jul 17, 2018
tests Hopefully finally fixing flaky Safari tests Jun 1, 2017
vendor Major reworking of routing, making for easier Toolforge and local setup May 31, 2018
views Collapse nav links under More menu if they don't fit Jul 17, 2018
.eslintrc Pageviews Analysis 2.0!!!!!!! Nov 12, 2016
.gitignore Major reworking of routing, making for easier Toolforge and local setup May 31, 2018
.scss-lint.yml Refactor date range validation Aug 21, 2016
.travis.yml Query and cache XTools page assessments config Apr 24, 2018
LICENSE Siteviews: rm inaccurate totals for unique devices, fix permalink issue Jan 15, 2018
README.md Fix occasional SQL bug in Pageviews API; fix i18n loading on local May 31, 2018
composer.json Pageviews Analysis 2.0!!!!!!! Nov 12, 2016
config.sample.php Fix FAQ and URL Structure pages, rm unneeded stylesheets May 31, 2018
deploy.sh Major reworking of routing, making for easier Toolforge and local setup May 31, 2018
gulpfile.js Fix FAQ and URL Structure pages, rm unneeded stylesheets May 31, 2018
haml.php Pageviews Analysis 2.0!!!!!!! Nov 12, 2016
nightwatch Acceptance tests!! Runs automatically with pushes/pull requests Mar 16, 2016
nightwatch.json add jquery.i18n for clientside translations May 5, 2016
package-lock.json Compiled assets for PR #195 Aug 29, 2018
package.json Compiled assets for PR #195 Aug 29, 2018
site_map.json Add missing wikis, wikimediafoundation.org -> foundation.wikimedia.org Sep 25, 2018
symlinks.sh Major reworking of routing, making for easier Toolforge and local setup May 31, 2018

README.md

Pageviews Analysis

A suite of tools to visualize pageviews data of Wikimedia Foundation wikis

Live tool: https://tools.wmflabs.org/pageviews (and Langviews, Topviews, Siteviews, Massviews, Redirect Views, Userviews, Mediaviews)

IE10 and Safari 8 and below are not supported.

Build Status

License

The source code for Pageviews Analysis and it's sister applications is released under the MIT license.

The underlying data shown in these applications was provided by the Wikimedia RESTBase API, released under the CC0 1.0 license dedication.

Screenshots of the charts in Pageviews Analysis may be used without attribution, but a link back to the application would be appreciated.

Dependencies

This guide mostly assumes you're using MacOS or Linux. The setup process may differ on Windows machines.

  1. Node

  2. PHP 5.6+ – PHP is for rendering partials in the views and for the Intuition i18n framework, and for some internal APIs that query the replicas.

  3. Composer

  4. ESLint and SCSS-lint – The linters are not necessary, but preferred so your code maintains a consistent style.

Setup

  1. Install all node packages and dependencies with npm install.

  2. Run composer install to install all PHP dependencies.

  3. Create a copy of config.sample.php named config.php. See the comments within that file on what you need to configure.

  4. Optionally create the "meta" database on your local machine. This database is used for usage tracking, and also to store known false positives for Topviews. Unless you're working on Topviews, you can skip this step.

  5. Run gulp to watch the javascripts, stylesheets and views and automatically recompile when new changes are saved.

  6. Start the server with php -S localhost:8000

You should be able to access each application as a subpath of localhost:8000, e.g. http://localhost:8000/pageviews.

Code walkthrough

This app aims to be a part of the future and not linger in the past. JavaScript is written in ES6 where possible, and transpiled to ES5 with Babel. If you need to add a polyfill for something, add it to /javascript/shared/polyfills.js.

All assets and views are ultimately placed in public_html. With the exception of images, you won't need to make any manual modifications to this directory.

Structure

The repo contains eight separate applications that share code with each other (Pageviews, Langviews, Topviews, Siteviews, Massviews, Redirect Views, Userviews, and Mediaviews). Each app has a dedicated subdirectory within javascripts, stylesheets and views. The main development asset files share the same name as the app (e.g. pageviews.js for the main JavaScript file for Pageviews). After compilation each app has it's own application.js and application.css in the public_html directory. PHP partials and Sass imports are prepended with underscores (e.g. _footer.php, _mixins.scss).

There are symlinks all throughout the public_html directory. These are to ensure you can use the native PHP server on your local machine and also deploy to Toolforge with the same code. The symlinks effectively change the document root of Toolforge's lighttpd server.

JavaScripts

Browserify is used to help follow a module pattern. You'll need to require('./file_name') any file that is a dependency. All JavaScript is written in ES6 (and possibly ES7).

Each app has it's own config.js, which are constants for application-wide use. Pageviews, Siteviews and Mediaviews also have a templates.js file that defines how data is shown in the right column on the interface.

Shared JavaScript goes in the /shared directory and must be required as needed. list_helpers.js is used on apps that have a list view, which are Massviews, Langviews, Redirect Views and Userviews. All apps but Topviews show a chart of some sort, and require chart_helpers.js.

When the JS files are compiled, they are concatenated into a single application.js that lives within the directory for that app inside public_html.

All JavaScript is documented using JSDoc. The documentation is hosted at https://tools.wmflabs.org/pageviews/jsdocs. You can generate the docs locally by running:

gulp jsdoc && open jsdocs/gen/index.html

Styles

Styles are written in http://sass-lang.com/ and compiled to CSS. The CSS is then automatically vendor-prefixed to support modern browsers. Bootstrap is used as the CSS framework.

Each page has it's own .scss file that imports dependencies. Shared files are simply prefixed with an underscore. _mixins.scss is for the mixins, placeholders, keyframes and colour variables. Similar to the JavaScripts, _list_view.scss is imported by apps that have a list view (Massviews, Langviews, Redirect Views and Mediaviews), and _chart_view.scss by apps that show charts.

Haml

The views within /views are written in MtHaml and compiled to PHP files in /public_html.

Local

Run gulp to watch for changes and automatically compile as needed. You can also run tasks by app and function, such as gulp massviews for all Massviews-related files, or gulp scripts to compile only the JavaScripts, but gulp by itself should be all you need.

Production

Before making a pull request or pushing to master, remember to run gulp production so the assets are minified and concatenated. JSDocs are also generated, and placed in a dedicated sub-repo (not submodule) in jsdocs/gen. These can optionally be pull requested to the JSDocs repo.

For deployment instructions, see Tool:Pageviews on Wikitech.org.

Tests

Tests will be ran automatically when you push or create a pull request.

Acceptance

Pageviews Analysis was once tested with acceptance tests, but after significant maintenance burden and general flakiness of the tests, they have been retired :(

Linters

The styling of all ES6 and SCSS is enforced with linters. You can run these locally with gulp lint, and will also be ran when you run gulp production. If you need a particular rule to be ignored, you can add exceptions (see Scss-lint, Eslint).