A CKAN extension to create visualizations based on the uploaded datasets.
- Creating new releases
- Development
- Custom Development Within the ckanext-querytool Extension
- Google Analytics (GA4)
Before creating a new release, see the next section, Test data seed command
On a freshly installed portal, with nothing but an admin user, you can run a paster command to populate the portal with test data found on the STEPS Data site. This includes the STEPS Survey organization, 9 datasets with 2 resources each, and 9 reports with multiple visualizations.
Before creating a new release, this command should be run to verify that there are no compatibility issues. This could be especially critical if there are any changes to the DB tables or structure.
To populate the portal, run:
paster --plugin=ckanext-querytool seed_portal -c /path/to/.ini
It will then prompt the user to enter the admin name and admin API key:
Enter the admin login name: <ADMIN_NAME>
Enter the admin API key: <API_KEY>
That's it! You'll see output for each step (but the overall run time is very fast) and it will alert you if it runs into any issues.
If you have issues running the command successfully, you might need to run these paster commands:
WARNING: The following commands will completely clear the DB, including the admin user, password, and email address. If you've created anything on the portal, you'll need to do so again.
paster --plugin=ckan db clean -c /path/to/.ini
paster --plugin=ckan db init -c /path/to/.ini
After you've recreated the admin user, run the seed_portal command again, with this admin user name and new API key.
TODO: Possibly add a make command to the ckan-cloud-docker Makefile.
NOTE: There's not a current full Docker environment for this project. The following steps are for setting up a source install of CKAN, but using a Docker image for Solr (See point 2 for using the Docker image for Solr) instead of a local install.
- Install CKAN 2.7.3 from source by following the steps found in the CKAN install documentation (Before proceeding, read through the following points).
- Make sure to install the tag
ckan-2.7.3. So, replacepip install -e 'git+https://github.com/ckan/ckan.git@ckan-2.7.12#egg=ckan'from part c. in the docs withpip install -e 'git+https://github.com/ckan/ckan.git@ckan-2.7.3#egg=ckan'. - Apply this patch to CKAN (fixes errors with datastore):
diff --git a/ckanext/datastore/helpers.py b/ckanext/datastore/helpers.py index b616f0f94..b58cb1a76 100644 --- a/ckanext/datastore/helpers.py +++ b/ckanext/datastore/helpers.py @@ -105,7 +105,12 @@ def get_table_names_from_sql(context, sql): table_names = [] try: - query_plan = json.loads(result['QUERY PLAN']) + if isinstance(result['QUERY PLAN'], list): + result_query_plan = json.dumps(result['QUERY PLAN']) + query_plan = json.loads(result_query_plan) + else: + query_plan = json.loads(result['QUERY PLAN']) + plan = query_plan[0]['Plan'] table_names.extend(_get_table_names_from_plan(plan)) - Update
psycopg2in the CKANrequirements.txtfile to this version before installing:psycopg2==2.7.3.2 - For the best compatibility, install PostgreSQL 12 if possible.
- Make sure to install the tag
- Clone and use this Solr docker-compose setup instead of a local Solr install (you can clone this in the same parent directory where you cloned CKAN. Using this option for Solr requires installing
dockeranddocker-composeon your OS).- Update
solr_urlin your.inifile (from step 1) if it differs from this:solr_url = http://127.0.0.1:8986/solr/ckan
- Update
- Clone and install DataPusher following the steps found in the documentation (you can clone this in the same parent directory where you cloned CKAN and Solr, but install it in a new virtual environment to avoid potential
SQLAlchemyversion conflicts with CKAN):- Update
ckan.datapusher.urlin your.inifile (from step 1) if it differs from this:ckan.datapusher.url = http://127.0.0.1:8800/ - Add
datastoreanddatapushertockan.pluginsin your.inior.envfile (in case you missed this step in the CKAN and DataPusher docs):ckan.plugins = querytool datastore datapusher EXTENSION_4 ...
- Update
- Clone and install this repo (
ckanext-querytool) (in thesrcdirectory of the same parent directory as CKAN and Solr, using the same Python virtual environment from step 1):# Installation (inside the CKAN virtual environment) cd ckanext-querytool python setup.py develop pip install -r requirements.txt- Add
querytoolto yourpluginsin your.inior.envfile.ckan.plugins = querytool EXTENSION_2 EXTENSION_3 ...
- Add
Once you've followed the above steps and everything is installed and ready, you can start the environment. This requires running a few commands, each in their own terminal.
- Start Solr:
# Running Solr (from the parent directory) docker-compose -f docker-ckan-solr/docker-compose.yml up - Start DataPusher:
# Running DataPusher (from the parent directory, inside the CKAN virtual environment) python datapusher/datapusher/main.py datapusher/deployment/datapusher_settings.py - Start CKAN (depending on how/where you set CKAN up, you might not need to use
sudo):# Running CKAN (from the parent directory, inside the CKAN virtual environment) sudo paster --plugin=ckan serve /PATH/TO/YOUR/INI_FILE
For building static assets, Node.js is required and can be installed with these commands:
$ nvm install 10
$ nvm use 10
$ npm installPut your scripts/fonts/etc inside the ckanext/querytool/fanstatic folder and images inside the ckanext/querytool/public folder. It can be used as usual ckan fanstatic and public contents. At the same time, we use CSS preprocessor (LESS) to build our styles. Put your styles inside the ckanext/querytool/fanstatic/less and build it:
$ make assetsProcessed styles will be put to the ckanext/querytool/fanstatic/css folder.
.babelrc -- babel config (see https://babeljs.io/docs/en/configuration#babelrc for config options)
webpack.config.js -- webpack config (see https://webpack.js.org/configuration/)
Source files for the querytool live in:
ckanext/querytool/fanstatic/javascript
Distribution files (files served via our extension) are compiled via webpack and live in:
ckanext/querytool/fanstatic/javascript/dist
NOTE: Make sure to reference /dist/ files, not the source files!
Vendor files (third party libraries) live in:
ckanext/querytool/fanstatic/javascript
Source files can be changed as needed. After updating run:
npm install && npm run webpack which will build the transpiled / minified resources and copy them to the /dist folder in the host machine.
To extract i18n messages and compile the catalog we have to have our development server running. In another terminal window run a command:
$ make i18n
See CKAN documentation for more on i18n management.
For the country flags to appear in the language selector it's needed to make sure there's an associated flag for each one of the configured languages. This must be done by further expanding the /ckanext-querytool/ckanext/querytool/public/base/resources/lang-flags.json file. Here's an example of how this file looks like:
{
"en": "gb",
"pt_BR": "br",
"es": "es",
"fr": "fr",
"km": "kh",
"zh_CN": "cn"
}
The property name is always the language value. The value is always the ISO 3166-1-alpha-2 code of the country.
To update this readme's table of contents run:
$ make readmeThe app allows to configure different layouts for an application's visualizations. We can test expected behavious in half-automated way using Chrome console in the brower:
- Go to /querytool/public/detailed-mortality-by-cause
- Copy-past and start in the console script from
bin/test-layouts.js
The extension supports some optional configurations:
- OpenStreetMaps
- Cookie Control
- reCAPTCHA v2 on login page
The example values below are the defaults. To override them, add the following to your .ini or .env file and change the values as needed:
ckanext.querytool.map_osm_url = https://cartodb-basemaps-{s}.global.ssl.fastly.net/light_nolabels/{z}/{x}/{y}{r}.png
ckanext.querytool.map_osm_attribute = © <a href="http://www.openstreetmap.org/copyright">OpenStreetMap</a> © <a href="http://cartodb.com/attributions">CartoDB</a>
For example, if you want to use a dark CartoDB basemap instead of the default light map, you can set the variables to the following:
ckanext.querytool.map_osm_url = https://cartodb-basemaps-{s}.global.ssl.fastly.net/dark_nolabels/{z}/{x}/{y}{r}.png
The second variable is used for the attribution/copyright text. For example, if you decide to use OpenStreetMap without CartoDB, you can set the variables to the following:
ckanext.querytool.map_osm_url = https://tile.openstreetmap.org/{z}/{x}/{y}.png
ckanext.querytool.map_osm_attribute = © <a href="http://www.openstreetmap.org/copyright">OpenStreetMap</a>
NOTE: Changing the map provider, though possible, is not recommended. If the alternative isn't a complete and identical drop-in replacement, the map will most likely not work as expected, and you will need to make changes to the code to accommodate the new map provider.
The Cookie Control panel can be configured using the following options:
ckanext.querytool.cc.enabled = True
ckanext.querytool.cc.api_key = your_api_key
ckanext.querytool.cc.license_type = COMMUNITY
ckanext.querytool.cc.popup_position = LEFT # Or RIGHT
ckanext.querytool.cc.theme_color = DARK # Or LIGHT
ckanext.querytool.cc.initial_state = OPEN # Or CLOSED
By default, if enabled is true, only api_key is required for the Cookie Control to work. Currently, it considers auth_tkt as a core cookie and Google Analytics Cookies as optionals. An API key can be generated by registering here.
It's possible o enable reCAPTCHA v2 on the login page by setting the following environment variables:
# Example values
ckanext.querytool.recaptcha.sitekey = 6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI
ckanext.querytool.recaptcha.secretkey = 6LeIxAcTAAAAAGG-vFI1TnRWxMZNFuojJ4WifJWe This extension uses LESS for styles. All changes must be made in one of the LESS
files located in the ckanext-querytool/ckanext/querytool/fanstatic/less folder.
Gulp.js is used for building CSS assets.
In order to build all CSS assets run node_modules/gulp/bin/gulp.js once. Gulp.js is watching for changes in the source LESS assets and will rebuild CSS on each change. If you have gulp.js installed globally on the system, you can just type gulp to run it.
This extension diverges from CKAN core in a number of ways. The following sections cover the major areas of divergence and areas most commonly customized, along with high-level descriptions and links to the relevant code and external documentation.
The frontend libraries and frameworks used by this extension for charts and tables are:
- Plotly for charts (bar, line, pie, etc.)
- DataTables for tables
- Leaflet for maps
These are used on top of:
If any changes are made to the JavaScript files, you must rebuild using webpack (see the Updating source files section above).
For more information on CKAN core architecture as a whole, see CKAN code architecture. For anything not covered here, see the CKAN 2.7 core documentation.
This extension uses plotly.js for charts (bar, line, pie, etc.). It's a JavaScript library for creating interactive charts and graphs, built on top of d3.js and stack.gl.
Most changes and customizations of plotly.js are done in the ckanext-querytool/ckanext/querytool/fanstatic/javascript/modules/viz-preview.js file.
The main settings object can be found in the viz-preview.js file, and it's called base_info. It contains the following properties:
var base_info = {
margin: {
l: 20,
r: 20,
b: 20,
t: 30,
pad: 5,
},
title: titleVal,
showlegend: show_legend, //show legend value
legend: {
xanchor: "left",
x: -0.02,
y: -0.27,
orientation: "h",
title: {
text: legend_title_text,
side: "top",
// Reference: https://plotly.com/javascript/reference/layout/#layout-legend
font: {
size: 15
}
}
},
xaxis: {
tickformat: x_tick_format,
automargin: true,
title: "",
tickangle: this.options.x_text_rotate,
tickmode: "auto",
nticks: this.options.x_tick_culling_max,
tickfont: {
size: 14,
},
categoryorder: "array",
categoryarray: sortedArr
},
yaxis: {
tickformat: y_tick_format,
automargin: true,
hoverformat: format,
tickangle: this.options.y_text_rotate,
tickfont: {
size: 14,
},
},
hovermode: "closest",
};
Any of these properties can be modified to change the default behavior of the chart. For example, if you want to change the default font size of the chart, you can change the size property of the tickfont object. Additionally, each chart type has its own set of properties that can be modified by looking for the respective trace object. For example, each line chart type has the following properties in their trace objects:
var trace = {
x: x,
y: columns[tmp].slice(1),
type: "scatter",
mode: labelsMode,
text: convertedTextTitles,
textposition: "top right",
textfont: {
size: 14,
},
name: name,
line: {
width: setDefaultWidth(lineWidthsList[tmp]),
dash: lineTypesList[tmp],
},
hovertemplate: "%{y}<extra></extra>",
error_y: {},
error_x: {},
};
NOTE: Though all of the properties are customizable, it's recommended to only change them if absolutely necessary, as any changes can have unintended consequences.
For more information on how to customize plotly.js, see the plotly.js documentation.
This extension uses DataTables for table charts/visualizations. DataTables is a JavaScript library for creating interactive tables, built on top of jQuery.
Changes and customizations of DataTables can generally be handled in ckanext-querytool/ckanext/querytool/fanstatic/javascript/vitals.js or ckanext-querytool/ckanext/querytool/fanstatic/javascript/modules/table-module.js.
An example of custom changes can be found at the very top of vitals.js, where a feature plugin (RowsGroup) is used to automatically merge columns cells based on their values eqaulity.
For more information on how to customize DataTables, see the DataTables documentation.
This extension uses Leaflet for map charts/visualizations. Leaflet is a JavaScript library for creating interactive maps, built on top of OpenStreetMap.
Changes and customizations of Leaflet can generally be handled in ckanext-querytool/ckanext/querytool/fanstatic/javascript/modules/map-module.js.
Most general jQuery and JavaScript customizations/overrides can be handled in ckanext-querytool/ckanext/querytool/fanstatic/javascript/vitals.js. Here's a small example where we watch for changes in the filter dropdowns and trigger a click on the "Update" button when a change is detected:
$("body").on('change','.has-filter .filter-item-value', function() {
$('.btn-update').trigger("click");
})
For more information on how to customize jQuery and JavaScript, see the jQuery documentation and JavaScript documentation.
For most changes to the options found in the UI (for charts and tables, e.g. https://YOUR_SITE/report/edit_visualizations/REPORT_NAME), you can find the relevant code in the following files:
CKAN (and this extension) uses Jinja2 for templating. The templates for this extension can be found in ckanext-querytool/ckanext/querytool/templates. Any changes in these HTML files should be reflected immediately in the UI immediately, but if you're not seeing them, you may need to restart CKAN.
Additionally, if you need to override a template found in CKAN core, you can simply copy the file found in CKAN core into the ckanext-querytool/ckanext/querytool/templates directory and make your changes there. You must make sure that it's placed in the same directory structure as the original file, e.g. if you want to override the package/read.html template, you would copy the file from ckan/templates/package/read.html to ckanext-querytool/ckanext/querytool/templates/package/read.html.
For more information on how to customize Jinja2 templates, see the Jinja2 documentation, CKAN Jinja2 documentation, and CKAN theming documentation.
The process for creating/customizing the API endpoints (Actions) is the slightly different than the CKAN documentation, but this is only because we have automated get_actions in plugin.py to return all actions found in ckanext-querytool/ckanext/querytool/logic/action (in this directory, we have 4 files: create.py, delete.py, get.py, and update.py, all of which are included in the automated loading), so they don't need to be added individually. We do this with the following code (found in plugin.py:
module_root = 'ckanext.querytool.logic.action'
action_functions = h._get_functions(module_root)
As a basic example, let's say we want to create a new GET endpoint that simply returns the text "This is our new API endpoint!".
- We would simply add the following code to
ckanext-querytool/ckanext/querytool/logic/action/get.py:@toolkit.side_effect_free def our_new_action(context, data_dict): return 'This is our new API endpoint!' - Restart CKAN
- Visit https://YOUR_SITE/api/3/action/our_new_action, and you should see the following response:
{ "help": "https://YOUR_SITE/api/3/action/help_show?name=our_new_action", "success": true, "result": "This is our new API endpoint!" }
Helpers are re-usable functions. They're available in the backend Python code as well as in Jinja2 templates. There's basically no limit to what a helper can do. They can range from simple functions that return a string, to more complex functions that parse, filter/alter, and return data.
The process for creating/customizing helpers is the same as found in the CKAN helper documentation.
As a basic example, let's say we want to create a new helper that simply returns the text "This is our new helper!".
- We would start by adding the following code to
ckanext-querytool/ckanext/querytool/helpers.py:def our_new_helper(): return 'This is our new helper!' - Add our new helper to
get_helpersinplugin.py:def get_helpers(self): return { ... OTHER HELPERS ... 'our_new_helper': helpers.our_new_helper } - Call the new helper in another function or in a template. For example, in a template, we can call it with:
{{ h.our_new_helper() }} - Restart CKAN
Now, you should see the text "This is our new helper!" in the template wherever you called it (or you can add a log in the backend function you called it from to see the output in the logs).
For a better idea of what helpers can be used for, you can take a look at the helpers in ckanext-querytool/ckanext/querytool/helpers.py and CKAN core helpers.
If you need to intercept or add new keys/values to the backend visualizations or reports objects, you can find much of the code in /ckanext/querytool/controllers/querytool.py (for example, edit_visualizations retrieves the saved values from the DB and returns them to the template) and /ckanext/querytool/model.py (this is where new DB columns are added).
To setup Google Analytics using GA4, you need to install a specific branch (ed-dev-ga4) of the Datopian Google Analytics extension:
ckanext-googleanalytics with GA4 support
Then follow the instructions in the "Setup" section of the README.