Skip to content

Home

Jump to bottom
fubits edited this page Mar 31, 2021 · 12 revisions

Documentation

-> Go back to README
-> Jump to JSON Dictionary
-> List of QC Steps

(Wiki Editing Tips)

  • a GitHub Wiki is a repository as well
    clone with git clone git@github.com:rankingdigitalrights/datalab.wiki.git

Introduction

The development environment (The development environment. Clockwise: VSCode+clasp; Google Apps Script; GitHub; Spreadsheet Output on Drive)

(top ↥)

To provide contributors with a cognitive model of this project, it is helpful to distinguish between

  • the development environment
    • collaboration & version control: source code shared on git, edited locally with an IDE
    • clasp for deploying to Google Apps Script,
  • the parameters & configuration: JSON files for Indicators, Companies, Research Steps, and global Index settings
    • the indicator & element language process has been stream-lined further with the introduction of the "indicator-factory" (cf. in @datapipe repo)
  • the execution / runtime environment (Google Apps Script online editor),
  • the output environment (Google Drive), and
  • the UI/frontend (currently Google Spreadsheets).
  • the data pipeline (in @datapipe repository), which is based on R, draws the data from the data stores, and produces all machine-readable outputs incl. curating the data for the Metabase SQL DB

Application Overview

(top ↥)

The basic structure is now modular, with

  • 1 Master controller 00_mainController.js in /* (root)
  • all core modules being in /src/* and beginning with 1x (DC), 2x (SC), etc., and
  • most parameters stored in /json/* (companies, indicators, research steps, global Index parameters)

With the exception of the Input Spreadsheets Module, all core modules have been split into ...

  • a main high-level module controller (x0)
  • a module interface (x1)
  • lower-level helper submodules (x2 - xN )

... in order to provide a single execution interface, to improve Code readability and re-use, and to enable affordable maintenance.

After modules had been introduced and eventually refactored, most modules and methods operate on the same centrally defined set of parameters (the JSON files) and the same set of helper functions (i.e. defining named Ranges). Therefore, if you make structural changes to e.g. researchSteps.step.component.labelShort in JSON_researchSteps.js while working on 10_inputInterface.js, you need to apply this structural change to all modules concerned (incl. Helpers) as it might impact e.g. 21_scoringInterface.js.

current best practice is to

  • stick to Clean Code principles (i.e. explicit and precise naming of variables / functions
  • use global search & replace with case-by-case replacement decisions

JSON: Parameters & Config

(top ↥)

currently stored in the Git repository as JSON/JS Objects (./json/JSON*.js)
TODO: should be offered either as clean JSON files (i.e. with fetch(JSON).parse()) or be imported from a central human-editable spreadsheet or AirTable config table

  • research steps
  • indicators
    • new: elemsMetadata for richtext formatting / optional hyperlinking of Company Feedback forms
  • companies
  • config

Research Steps

  • JSON_researchSteps.js

Indicators

TODO: currently, SSOT for indicator- and element-level **language **is in the @datapipe repository (specs-ssot/indicators.html). But SSOT for meta-**specifics **such as scoring scopes or legacy subcomponent relations are only defined in this repository
Therefore see Notion/Specs for current editing workflow.
TODO: proposal - if Language isn't going to be relocated to a central spreadsheet, then Language should be relocated to this repository and then both, Specs (json/JSON_Indicators.js) and Language being imported with a fetch() by @datapipe

  • JSON_indicators.js (Specs)
  • JSON_elemsMetadata.js (Tags / Links for element-level language)

TODO: eventually merge both into 1 file and adjust @datapipe parsers

Companies JSON

(top ↥)

caution: make sure that service children are listed in order of previous Index' listing (i.e. prepaid, postpaid, broadband)

  • Guidelines

    • Baseline: use 1-letter category prefix for companies and 2-letter prefixfor services
    • Authoritative: Dictionary
    • use a GUI JSON editor for validation of JSON syntax (i.e. VSCode + JSON Editor plugin, or JSON Editor Online)

VSCode with JSON Editor Plugin (Screenshot: VSCode with JSON Editor Plugin: JSON file is opened on the left, the GUI editor on the right.)

Config JSON

(top ↥)

defines Index, methodology / sets, companies, folders, and maybe maps editors to companies (should be done from central spreadsheet for convenience for research team)

Environment: Google Apps Script / Apps API

(top ↥)

Motivational quote:

The GAS documentation is not complete now and many features are discoverable experimentally." Source

Learnings

(top ↥)

  • if you update existing files, named ranges are not removed by sheet.clear()
  • max. runtime is 30 minutes / 1800 seconds. If your script breaks due to runtime error, you can see the log (Ctrl+Enter in the online editor) to see which company was processed last, delete the output for this company, and re-run the script with a subset, starting with the unfinished company.
  • also, you can run multiple scripts in parallel… For this, slice(1,4) the companies vector in multiple parts in 00_mainController.js
  • file name ambiguity --> use fileID where possible
  • only owner can delete files for real
  • cell-based operations are costly, with formatting being the most expensive ones. Try to use array-based methods (2D range [][]) as much as possible.
  • ...

To be explored

(top ↥)

Clone this wiki locally