Skip to content

Microdata Output

Jump to bottom
esseff edited this page Jan 13, 2023 · 123 revisions

Home > Model Development Topics > Microdata Output

Microdata output allows a model to output records containing the values of selected entity attributes during a run for later use. This topic describes microdata output from a model developer perspective.

Related topics

Topic contents

Introduction and outline

A model built with microdata output capability can output records containing the values of entity attributes. As well as attribute values, each microdata output record contains a microdata key to match corresponding records between runs.

By default, a model does not have microdata output capability. See Enabling microdata output or Quick start on how to build a model with microdata output capability.

Two microdata output modes are supported: text mode and database mode. Text mode is targeted more to model developers, while database mode is targeted more to users of production models and to future run-time tabulation functionality. Text mode writes microdata to one or more text files in csv format. Text mode supports run-time filtering based on event context, and can include an optional output column to provide event context. Database mode writes microdata to the model database, from which it can be extracted using dbcopy or a future API. Database mode will be used for future run-time tabulation functionality, including microdata comparisons between runs.

Microdata output is controlled by run-time settings, build-time settings, and model code.

Run-time settings specify which attributes are output during a run, provided the model was built with microdata output capability. All attributes are available for selection at run-time without rebuilding the model. Some run-time settings apply only to text mode. Those text mode settings can filter records by event context and can create an additional column showing the event context for each record.

Build-time settings are statements in model code which make the model capable of microdata output, and (optionally) when microdata output occurs in the entity life cycle: on entrance, on exit, or on the occurrence of an event.

Model code can write microdata explicitly by calling the supplied entity member function write_microdata. The write_microdata function can be hooked to an existing entity function such as the implementation function of an event.

Quick Start shows how to build a model capable of microdata output and how to activate that capability in a model run.

The quick start is followed by several worked examples with illustrative inputs and outputs, mostly using the RiskPaths model.

The first set of examples entity life cycle, entity life cycle with event context, and entity life cycle with event filtering illustrate how to probe the life cycle of entities using microdata text mode.

The second set of examples output using a hook to a model event, output using a hook to a self-scheduling attribute, and output by calling write_microdata in model code illustrate how to control when microdata output occurs from model code.

The final example illustrates database output in a time-based model including extracting microdata from the database using dbcopy.

The worked examples are followed by subtopics containing reference information:

[back to topic contents]

Quick start

This subtopic contains the following sections.

[back to topic contents]

1. Build model with microdata output capability

Add the following statements to the model source code file RiskPaths/code/ompp_framework.ompp:

options microdata_output = on;
options microdata_write_on_exit = on;

Build the Release version of RiskPaths.
In Windows, the model executable will be RiskPaths/ompp/bin/RiskPaths.exe.
In Linux, the model executable will be RiskPaths/ompp-linux/bin/RiskPaths.

[back to quick start]
[back to topic contents]

2. Modify model ini file with microdata output options

In the same folder as the RiskPaths executable there may already be a copy of the default model ini file RiskPaths.ini. If not create it using your IDE or a text editor such as Notepad.

Edit RiskPaths.ini to have the following content:

[Parameter]
SimulationCases = 5

[Microdata]
ToCsv = yes
Person = age, union_status, parity_status

[back to quick start]
[back to topic contents]

3. Run model using microdata output

Launch the model in its bin directory using the ini file created in the previous step.

RiskPaths -ini RiskPaths.ini

In Windows you can run the Release version of RiskPaths from inside Visual Studio as follows:

  • Solution Configurations to Release and Solution Platforms to x64
  • Project Properties > Configuration Properties > Debugging > Command Arguments to
    -ini RiskPaths.ini
  • Project Properties > Configuration Properties > Debugging > Working Directory to $(TargetDir)
  • To launch the model, do Debug > Start without debugging or Press Ctrl-F5.

When the model run completes, the file RiskPaths.Person.microdata.csv should be present in the model bin directory and look like this:

key,age,union_status,parity_status
1,100,2,1
2,100,2,1
3,100,2,1
4,100,0,1
5,100,0,1

or formatted as a table, like this:

key age union_status parity_status
1 100 2 1
2 100 2 1
3 100 2 1
4 100 0 1
5 100 0 1

The run-time settings output the attributes age, union_status, and parity_status. The leading column key can be used to match microdata records between runs. The build-time option microdata_write_on_exit causes a microdata record to be written whenever an entity leaves the simulation. In RiskPaths there is no mortality and Person entities exit the simulation at age 100. The values of union_status and parity_status are those at that age, for each Person entity in the run.

The model log contains the following warning, which is expected.

Warning : model can expose microdata at run-time with output_microdata = on

[back to quick start]
[back to topic contents]

Worked example 1a

This section continues the quick start example to output multiple microdata records for a single entity, when an entity enters and leaves the simulation, and at each event.

In ompp_framework.ompp, change the build-time microdata settings to

options microdata_write_on_enter = on;
options microdata_write_on_exit = on;
options microdata_write_on_event = on;

Change the run-time settings in RiskPaths.ini to consist of only one case

[Parameter]
SimulationCases = 1

[Microdata]
ToCsv = yes
Person = age, union_status, parity_status

and run the model.

Here's the resulting microdata output in RiskPaths.Person.microdata.csv, with some rows elided.

key age union_status parity_status
1 0 0 0
1 1 0 0
1 2 0 0
1 3 0 0
... ... ... ...
1 22.5 0 0
1 23 0 0
1 24 0 0
1 24.2609992115357 1 0
1 25 1 0
1 25.2609992115357 1 0
1 26 1 0
1 26.5378127283906 1 1
1 26.5378127283906 1 1
1 27 1 1
1 27.2609992115357 1 1
1 27.2609992115357 2 1
1 27.5 2 1
1 28 2 1
1 29 2 1
1 29.2609992115357 2 1
1 30 2 1
... ... ... ...
1 99 2 1
1 100 2 1
1 100 2 1
1 100 2 1

The microdata output shows the values of the attributes at every event in the life cycle. Multiple microdata records can occur at the same age due to multiple tied events at that age.

Worked example 1b

This example continues the previous example, outputting event context information for each microdata record.

Leave the build-time microdata settings in ompp_framework.ompp unchanged from the previous example:

options microdata_write_on_enter = on;
options microdata_write_on_exit = on;
options microdata_write_on_event = on;

Activate the CsvEventColumn option by modifying the run-time settings in RiskPaths.ini so that it looks like this:

[Parameter]
SimulationCases = 1

[Microdata]
ToCsv = yes
CsvEventColumn = true
Person = age, union_status, parity_status

Run the model.

Here's the resulting microdata output in RiskPaths.Person.microdata.csv, with some rows elided.

key event age union_status parity_status
1 (no event) 0 0 0
1 om_ss_event 1 0 0
1 om_ss_event 2 0 0
1 om_ss_event 3 0 0
... ... ... ... ...
1 om_ss_event 22.5 0 0
1 om_ss_event 23 0 0
1 om_ss_event 24 0 0
1 Union1FormationEvent 24.2609992115357 1 0
1 om_ss_event 25 1 0
1 om_ss_event 25.2609992115357 1 0
1 om_ss_event 26 1 0
1 FirstPregEvent 26.5378127283906 1 1
1 om_ss_event 26.5378127283906 1 1
1 om_ss_event 27 1 1
1 om_ss_event 27.2609992115357 1 1
1 UnionPeriod2Event 27.2609992115357 2 1
1 om_ss_event 27.5 2 1
1 om_ss_event 28 2 1
1 om_ss_event 29 2 1
1 om_ss_event 29.2609992115357 2 1
1 om_ss_event 30 2 1
... ... ... ... ...
1 om_ss_event 99 2 1
1 om_ss_event 100 2 1
1 DeathEvent 100 2 1
1 DeathEvent 100 2 1

The event column contains the name of the event being implemented when microdata output occurred. There is no event at the beginning of a case in a case-based model like RiskPaths, so when the first entity in the case enters the simulation (no event) is shown in the event column. If the event associated with microdata output is a self-scheduling event, om_ss_event is shown in the event column. Event Trace can be used to obtain more information about events, including the names of self-scheduling events.

The final three microdata output records all occur at age 100.

The first is from the self-scheduling event which maintains the derived attribute self_scheduling_int(age). That derived attribute is in turn used in the declaration of the identity attribute integer_age:

actor Person 					//EN Individual
{
	//EN Current integer age
	LIFE integer_age = COERCE( LIFE, self_scheduling_int(age) );
...

The second is from the event DeathEvent which is triggered by model logic and the ProbMort parameter immediately when integer_age is 100:

TIME Person::timeDeathEvent()				
{
    TIME event_time = TIME_INFINITE;
    if (CanDie)										
    {
        if (ProbMort[integer_age] >= 1) 			
        {
            event_time = WAIT(0);
        }
...

The third occurs when the entity leaves the simulation, because the option microdata_write_on_exit is on in the example. The event DeathEvent was the active event when the entity left the simulation, so that's what's shown in the event column.

Although it's not illustrated in this example, the name in the event column can be prefixed by a *. This indicates that the active event is in a different entity than the one being output. This can occur in a time-based model or in a case-based model with multiple entities in a case. For example a ChildBirth event in a Person entity could cause a new Person entity to enter the simulation and generate a microdata output record. The microdata record for the newborn would contain *ChildBirth in the event column to indicate that the active event was in a different entity than the microdata record.

[back to topic contents]

Worked example 1c

This example extends the previous example by filtering on specific events.

Leave the build-time microdata settings in ompp_framework.ompp unchanged from the previous example:

options microdata_write_on_enter = on;
options microdata_write_on_exit = on;
options microdata_write_on_event = on;

Modify the run-time settings in RiskPaths.ini to increase the number of cases to 5000, and restrict output to two named events using the Events option:

[Parameter]
SimulationCases = 1

[Microdata]
ToCsv = yes
CsvEventColumn = true
Person = age, union_status, parity_status
Events = Union1FormationEvent, FirstPregEvent

Run the model.

The resulting microdata output RiskPaths.Person.microdata.csv has 8,128 records and looks like this:

key event age union_status parity_status
1 Union1FormationEvent 24.2609992115357 1 0
1 FirstPregEvent 26.5378127283906 1 1
2 Union1FormationEvent 22.0523726276488 1 0
2 FirstPregEvent 24.6780778011483 1 1
3 Union1FormationEvent 17.050111243303 1 0
3 FirstPregEvent 20.024664717724 1 1
4 FirstPregEvent 17.4107170399441 0 1
5 FirstPregEvent 24.1577392012077 0 1
6 Union1FormationEvent 22.502915072767 1 0
6 FirstPregEvent 24.7534475294375 1 1
... ... ... ... ...

This csv file can be used to perform multivariate statistical analysis. For example, the csv file can be opened in Excel, the data can be filtered to just FirstPregEvent and a histogram generated to visualize the first birth distribution by age:

Age histogram of first births

The data could be filtered in Excel using the union_status column to visualize how union status affects the age distribution and produces the overall pattern.

[back to topic contents]

Worked example 2a

This example illustrates how to output microdata whenever a specific event occurs. Specifically, microdata will be output whenever the FirstPregEvent occurs in RiskPaths.

In RiskPaths, prepare the event implement function for hooks by adding the required statement at the end of the event implement function FirstPregEvent:

void Person::FirstPregEvent()
{
    parity_status = PS_PREGNANT;
    hook_FirstPregEvent();
}

Next, add code to hook the built-in function write_microdata to FirstPregEvent:

actor Person {
    hook write_microdata, FirstPregEvent;
};

In ompp_framework.ompp, turn off options which automatically write microdata, which were previously turned on in example 1.

//options microdata_write_on_enter = on;
//options microdata_write_on_exit = on;
//options microdata_write_on_event = on;

The statements inserted in example 1 were commented to revert to the default value off. This means that only explicit calls to write_microdata will generate microdata output.

Set the number of cases to 20 in RiskPaths.ini:

[Parameter]
SimulationCases = 20

[Microdata]
ToCsv = yes
Person = age, union_status, parity_status

Run the model.

The microdata output file RiskPaths.Person.microdata.csv should look like this:

key age union_status parity_status
1 26.5378127283906 1 1
2 24.6780778011483 1 1
3 20.024664717724 1 1
4 17.4107170399441 0 1
5 24.1577392012077 0 1
6 24.7534475294375 1 1
7 18.2797585879836 1 1
8 22.110326319997 1 1
9 21.2430736420085 1 1
10 29.168835553187 1 1
12 37.7955780112222 2 1
14 26.9550960057145 1 1
15 21.6012847802494 0 1
16 20.3178392448776 1 1
18 22.8298415328563 1 1
19 26.7999269606788 1 1
20 19.0257883348614 1 1

The microdata file shows the values of attributes at all occurrences of the FirstPregEvent in the run. It could, for example, be used to chart the distribution of age at first birth using a downstream application like Excel or R.

[back to topic contents]

Worked example 2b

This example shows how to produce microdata output using a self-scheduling attribute.

Change the hook in the previous example to

actor Person {
    hook write_microdata, self_scheduling_int(age);
};

and simulate a single case by modifying RiskPaths.ini:

[Parameter]
SimulationCases = 1

[Microdata]
ToCsv = yes
Person = age, union_status, parity_status

Run the model. Microdata output should look like this:

key age union_status parity_status
1 1 0 0
1 2 0 0
1 3 0 0
1 4 0 0
... ... ... ...
1 26 1 0
1 27 1 1
1 28 2 1
1 29 2 1
... ... ... ...
1 100 2 1

The microdata output contains a snapshot of the attributes at each integer age.

[back to topic contents]

Worked example 2c

under construction Output by calling write_microdata in model code

[back to topic contents]

Worked example 3

Under constructionDatabase output in a time-based model This example outputs microdata containing a population snapshot for the time-based model IDMM.

[back to topic contents]

Microdata output modes

Under construction

[back to topic contents]

Enabling microdata output

A model is capable of writing microdata if and only if model code contains the following statement:

options microdata_output = on;

A model with microdata capability will write the following warning to the log whenever it is run:

 Warning : model can expose microdata at run-time with microdata_output = on

If this is not a concern, for example if the model generates entities synthetically, this warning can be disabled by the following statement:

options microdata_output_warning = off;

A weight-enabled model which is also microdata-enabled will write the following message to the log when run

Note : model is weight-enabled and microdata-enabled, include entity_weight in Microdata for downstream weighted operations

as a reminder that the attribute entity_weight needs to be included in microdata output for downstream weighted tabulation.

Some internal entity attributes are created by the OpenM++ compiler. For example, the compiler creates an identity attribute to implement the filter of an entity table. These internal entity attributes are normally hidden. They can be made visible, including as microdata, using the following statement:

options all_attributes_visible = on;

[back to topic contents]

Run-time settings

Run time settings are specified as options, on the command line or in an ini file. Below are commented examples of all run-time microdata settings:

[Microdata]

; Person = ageGroup,sex,age,income,isOldAge,pension

; Store all non-internal attributes of Person entity
;
; Person = All

; Store all non-internal attributes of all entities
; NOT recommended for production, use for debug only
;
; All    = true

; Allow to store entities internal attributes
; NOT recommended for production, use for debug only
;
; UseInternal = true

; Write microdata entity attributes into database
; Important: each microdata entity MUST have unique key
;
; ToDb  = false

; Write microdata entity attributes and events (if enabled) into CSV file(s)
; each microdata entity is written in it's own file
;
; ToCsv = false

; Write microdata entity(s) attributes and events (if enabled) into model Trace output
; Trace output must be enabled to produce any results;
; see Trace options in [OpenM] section above
;
; ToTrace = false

; Write selected events into Trace or CSV file
;
; Events = Birth,Union,Death

; If true then write event name into CSV file
;
; CsvEventColumn = true

[back to topic contents]

Build-time settings

Build-time options determine when microdata is written during the simulation of each entity. The available options are:

Option Default Description
microdata_write_on_enter off microdata is written when the entity enters the simulation, before any event occurs in the entity.
microdata_write_on_exit off microdata is written when the entity exits the simulation.
microdata_write_on_event off microdata is written after an event occurs in the entity.

These options can be combined. If none of these options are on no microdata will be written unless model code does so explicitly by calling or hooking the built-in function write_microdata.

Note that attributes of an entity can change due to events in other linked entities in a model with interacting entities. So, even if these options are on, changes in attributes of an entity can be absent from microdata output for that entity. To see all changes in attributes in a model with interacting entities, consider using Event Trace.

[back to topic contents]

Writing microdata from model code

Microdata can be written by calling the built-in entity member function write_microdata() from model code, either directly or by using a hook statement.

[back to topic contents]

The microdata key

A key is a unique identifier used to match entities or microdata records across runs. It is a 64-bit value of C++ type uint64_t.

The key for an entity is returned by the entity member function get_entity_key. If this function is not defined in model code, the OpenM++ compiler will provide a definition which returns the value of entity_id.

The key for a microdata output record is produced by the entity member function get_microdata_key If this function is not defined in model code, the OpenM++ compiler will provide a definition which returns the value of the function get_entity_key.

The function xz_crc64 is provided to create a 64-bit key using the crc-64 open source checksum (hash) algorithm. The key can be based on a single value, or on multiple values, as illustrated in the following examples.

Here's a generic example using a location in memory, and a size in bytes.

uint64_t key = xz_crc64(ptrToBytes, sizeInBytes, 0); // calculate crc-64 for bytes array

Here's an example using multiple attributes of a Person entity:

uint64_t key = xz_crc64( &((uint8_t *)(Person->age)), sizeof(int), 0); // start new crc-64 key
key = xz_crc64( &((uint8_t *)(Person->income)), sizeof(double), key);  // continue crc-64 key calculation
key = xz_crc64( &((uint8_t *)(Person->isOld)), sizeof(bool), key);     // continue crc-64 key calculation

This example just illustrates how to create a key. A real example would use attributes which allow matching across runs.

[back to topic contents]

Home

Getting Started

Model development in OpenM++

Using OpenM++

OpenM++ user interface

Model Development Topics

OpenM++ web-service: API and cloud setup

Using OpenM++ from Python and R

Docker

OpenM++ Development

OpenM++ Design, Roadmap and Status

OpenM++ web-service API

GET Model Metadata

GET Model Extras

GET Model Run results metadata

GET Model Workset metadata: set of input parameters

Read Parameters, Output Tables or Microdata values

GET Parameters, Output Tables or Microdata values

GET Parameters, Output Tables or Microdata as CSV

GET Modeling Task metadata and task run history

Update Model Profile: set of key-value options

Update Model Workset: set of input parameters

Update Model Runs

Update Modeling Tasks

Run Models: run models and monitor progress

Download model, model run results or input parameters

Upload model runs or worksets (input scenarios)

Download and upload user files

User: manage user settings

Model run jobs and service state

Administrative: manage web-service state

Global Administrator: manage all web-services

Clone this wiki locally