Skip to content

Screened tables for Statistical Disclosure Control. 17 of February 2025

Choose a tag to compare

View all tags
@amc1999 amc1999 released this 17 Feb 21:32
· 84 commits to master since this release
v1.17.7
1673b3b

Screened tables for Statistical Disclosure Control. 17 of February 2025

v1.17.7

Main feature of this release is a Screened tables for Statistical Disclosure Control (SDC) and Output Quality Management.

Following included in this release:

  • c++ core: screened tables for statistical disclosure control
  • c++ core: new statistics in entity tables: mean, variance, stdev
  • c++ core: new API functions for model name and version
  • go: dget get list of model runs and worksets
  • UI: bug fix in download of model run results with deimension items IDs

New feature - Screened tables for Statistical Disclosure Control (SDC) and Output Quality Management

This new feature allows a model developer to implement model-specific SDC for outward-facing models which use confidential microdata. This feature can also be used to manage output quality, for example by limiting the number of digits of precision or by suppressing statistically unreliable cells. The new functionality applies at the sub/replicate/member level and at the individual table cell level. For more, please see our wiki topic

https://github.com/openmpp/openmpp.github.io/wiki/Screened-Tables.

New statistics in entity tables: mean, variance, stdev (+remark on support for statistics in weight-enabled models)

Three new statistics (aka accumulators) are now supported in entity tables: mean, variance, and stdev.

The mean(x) statistic is logically equivalent to the previous idiom sum(x)/unit.

For unweighted tables, mean(x) is computed using a numerically stable algorithm (the Welford algorithm) which is robust to floating point precision issues which can arise when many small values are added to a larger value.

For weighted tables, mean(x) is computed using the intuitive 'naïve algorithm', namely the quotient of the weighted sum of x divided by the sum of weights, i.e. the same algorithm as the idiom sum(x)/unit.

The previous idiom sum(x)/unit gives precise results for typical MSM use cases. For example, in a test, the mean of 500M income-like values using the numerically stable algorithm underlying mean(x) gave identical results to the 'naïvealgorithmsum(x)/unit`.

That said, using mean(x) instead of sum(x)/unit when an average is desired can mitigate a class of possible numerical issues (for unweighted models).

A numerically stable algorithm for mean for weight-enabled models might be implemented in a future OpenM version.

The mean(x) statistic can be particularly useful for Screened Tables because Screened Tables function at the level of accumulator/statistic (not expression), so may not work well for tables which use the idiom sum(x)/unit to compute averages.

The two other new statistics variance and stdev are also computed using a numerically stable algorithm. These statistics could also be computed using a 'naïve' one-pass algorithm which uses sums of squares and squares of sums. However, that naïve algorithm can be very unstable numerically because it suffers from a floating‑point phenomenon called ‘catastrophic cancellation’. The algorithm for variance and stdev used in OpenM does not suffer from that issue.

The statistics variance and stdev are not currently supported for weighted tables. That’s also true for the collection-based ordinal statistics P2, …, P50, …, P98, and gini. The OpenM compiler will emit an error if any of those statistics are used in a weighted table. However, an untransformed table in a weight‑enabled model is unweighted, so will be accepted by the compiler and computed as expected. A future version of OpenM might support weighted versions of these statistics.

New API functions for model name and version

This version contains 3 new functions which can be used in model code to obtain the model name and model version. Please note that the model name and model version are already visible in the UI and elsewhere. These functions can be used to meet special requirements from within model code, e.g. show a message about unimplemented functionality, or export the model version to a downstream model in a table.

Function Example
GetModelName() "GMM"
GetModelVersion() "2.0.1.0"
GetModelVersionInt() 2010

Get list of model runs and input scenarios from database

Dbget utility now allow to get list of model runs and input scenarios:

dbget -m modelOne -do run-list
dbget -m modelOne -do run-list -csv
dbget -m modelOne -do run-list -tsv
dbget -m modelOne -do run-list -json
dbget -m modelOne -do run-list -lang fr-CA
dbget -m modelOne -do run-list -dbget.NoLanguage
dbget -m modelOne -do run-list -dir my/output/dir
dbget -m modelOne -do run-list -f my-runs.csv
dbget -m modelOne -do run-list -pipe
dbget -m modelOne -do run-list -lang fr-CA -dbget.Notes

dbget -m modelOne -do set-list
dbget -m modelOne -do set-list -csv
dbget -m modelOne -do set-list -tsv
dbget -m modelOne -do set-list -json
dbget -m modelOne -do set-list -lang fr-CA
dbget -m modelOne -do set-list -dbget.NoLanguage
dbget -m modelOne -do set-list -dir my/output/dir
dbget -m modelOne -do set-list -f my-scenarios.csv
dbget -m modelOne -do set-list -pipe
dbget -m modelOne -do set-list -lang fr-CA -dbget.Notes

For more information please check Dbget Go documentation

Download source code and binaries:

Download release documentation archive:

Download cluster version (MPI):

IMPORTANT:
Full version of OpenM++ source code always included into "Download" links above.
Please do NOT use "Source code (zip)" or "Source code (tar.gz)" archives from "Assets" links below.
It is auto-generated by GitHub release tools and contains only half of OpenM++ source code.

Assets 12
Loading