Skip to content

2020-Q2
Compare
Choose a tag to compare
@shapiromatron shapiromatron released this 25 Jan 14:37
· 748 commits to main since this release
2020-Q2
51cccc2
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Released July 15 2020.

This quarter's updates focused on developing new visualizations for displaying data extracted in HAWC or in other tools. Evidence maps created in Tableau can now be embedded into HAWC, or if data has been extracted in HAWC, exploratory heatmaps can be created to view the data. In addition, standardized, commonly used views of data in an assessment are available whenever data has been extracted into HAWC (eg., see the dashboards in PFBS). These new views are available on all new and existing assessments.

Updates were also made in expanding data-capture and display of study evaluation/risk of bias data. In addition, a new assessment-level Dataset data type was added to allow users to upload datasets created in external systems (such as DistillerSR, SWIFT, or Excel) and use in visualizations in HAWC. We anticipate other features will be available with the new Dataset type in future updates.

We've also expanded the API and HAWC client so that studies, study evaluation/risk of bias responses, and animal bioassay data can be created via API in addition to our standard exports of these data. This should allow for easier interoperability with external datasets, databases, and applications.

A full list of changes are available below:

  • Assessment

    • Add new model, Dataset, where tabular datasets can be uploaded and used in HAWC assessments. Datasets are versioned and only the most recent version is available to the public if an assessment is made publicly available. Datasets can currently be used in creating custom exploratory heatmaps, but it is expected that they will be used in the future for other purposes.
    • Added a new assessment option to include data extracted from assessment in an anonymized machine learning (ML) dataset export
    • beta-testing: Added new ML exports using HAWC data

  • Literature

    • Previously, only references tagged with "Included" or child-tag could be extracted in HAWC; this is now a configurable option to allow for more dynamic tag-trees
    • Integration of literature tags with new exploratory heatmap visualizations
    • beta-testing: topic modeling for literature using title and abstract and visualization of topics. Preliminary approach using TFIDF + NMF + TSNE with the scikit-learn library.

  • Study evaluation/Risk of bias

    • Add new (optional) field to all risk of bias/study evaluation - direction of effect
    • Update visualizations to include the direction of effect when entered

  • Visualization

    • Add new custom visualization type - exploratory heatmap. Exploratory heatmaps can be created with data currently extracted in HAWC (literature tags, bioassay extraction, or epi extraction) or with a tabular dataset upload
    • In addition to custom exploratory heatmaps, standard generic heatmap visuals are available for extracted HAWC data. These are preloaded using commonly-used displays of bioassay and epi data (eg., PFBS), and are available for all assessments that have data extracted.
    • Add new visualization type - embedding external visuals. This allows embedding of visuals from external systems; it currently supports the public Tableau website.
    • Data pivot updates:
      • Update user interface (UI) for conditional formatting options in data pivots
      • Fix bug regarding column text alignment with maximum text-size
      • Fix bug, remove dose-groups where data are not extracted for a single Endpoint, even if it was captured in the Dosing Regime
    • Crossview updates:
      • Add endpoint name as a new filter option
    • Study evaluation/risk of bias heatmap
      • Update legend to include footnotes for additional extraction indications
      • Update heatmap to display direction of effect if evaluated
    • beta-testing: new visualization for exploring critical effect levels (LOEL, NOEL, BMD, etc) across all endpoints in an assessment, grouped by system

  • Client

    • Release python client on pypi to allow for easier install (eg., pip install hawc_client)
    • Update python client to create study, risk of bias response, or bioassay data
    • Add a status-bar indicator for large dataset downloads
    • Added a test-suite to Python client for continuous integration testing

  • API

    • Add new endpoints to create risk of bias response
    • Add new endpoints to create study
    • Add new endpoints to create animal bioassay extraction
    • Add new tabular exports, commonly used in exploratory heatmap visuals
    • Add new assessment dataset API views
    • Add OpenAPI API schema definition file (admin-only)

  • Testing

    • Add vcrpy to test suite for faster tests and to remove dependency on external websites like PubMed, HERO, etc.
    • Add new make coverage command to create testing coverage reports
    • Add test-suite for hawc_client and integrate with CI
    • Expanded tests for tabular exports to improve stability, commonly used in visualizations

  • Other

    • Refactor tabular exports to use REST API instead of custom solution
    • Remove printf in javascript in favor of ES6 string literals
    • Large tabular datasets are now cached and responses are reused over a period of hours. If data changes overtime, users now have the option to delete assessment caches and build new.
Assets 2