Microdata Output

Home > Model Development Topics > Microdata Output

Microdata output allows a model to output values of entity attributes during a run for later use. This topic describes microdata output from a model developer perspective.

Related topics

• Model Code
• Event Trace

Topic contents

• Introduction and outline
• Quick start How to build and run a model with microdata output
• Worked example 1 Entity lifecycle
• Worked example 2 hook to model event
• Worked example 3 hook to self-scheduling attribute
• Worked example 4 population snapshots in time-based model
• Enabling microdata output
• Run-time settings Run-time settings
• Build-time settings Build-time settings
• Model code Controlling microdata output from model code
• Microdata key Setting the microdata key

Introduction and outline

A model with microdata output enabled can output values of entity attributes during a run for later use. Output can be .csv files or internally in the run database.

Microdata output is controlled by run-time settings, build-time settings, and model code.

Run-time settings specify which attributes (if any) are output during a run. If a model is enabled for microdata output by build-time settings, all attributes are available for selection at run-time without rebuilding the model.

Build-time settings are statements in model code which enable a model to output microdata and can specify when microdata is written during the entity life cycle (enter, exit, event).

Optionally, model code can write microdata explicitly by calling the write_microdata() function, or by hooking write_microdata to an existing entity function in model code such as the implementation function of a specific event.

During a run, microdata can be written to the model database, to a text file in csv format, or to both.

A future version of dbcopy will allow extraction of run microdata from a model database for downstream processing. A future OpenM++ utility will allow tabulation of run microdata in the model database, including micro-level comparison operations between runs.

The examples in this topic use the csv output format.

Quick Start shows how to build a model capable of microdata output and how to activate that capability in a model run.

The quick start is followed by several worked examples with illustrative inputs and outputs using the RiskPaths model:

The worked examples are followed by several subtopics containing reference material:

[back to topic contents]

Quick start

This subtopic contains the following sections.

• 1. Build model with microdata output capability
• 2. Create model ini file with microdata output options
• 3. Run model using microdata output

[back to topic contents]

1. Build model with microdata output capability

Add the following statements to the model source code file RiskPaths/code/ompp_framework.ompp:

options microdata_output = on;
options microdata_write_on_exit = on;

Build the Release version of RiskPaths.
In Windows, the model executable will be RiskPaths/ompp/bin/RiskPaths.exe.
In Linux, the model executable will be RiskPaths/ompp-linux/bin/RiskPaths.

[back to quick start]
[back to topic contents]

2. Modify model ini file with microdata output options

In the same folder as the RiskPaths executable there may already be a copy of the default model ini file RiskPaths.ini. If not create it using your IDE or a text editor such as Notepad.

Edit RiskPaths.ini to have the following content:

[Parameter]
SimulationCases = 5

[Microdata]
ToCsv = yes
Person = age, union_status, parity_status

[back to quick start]
[back to topic contents]

3. Run model using microdata output

Launch the model in its bin directory using the ini file created in the previous step.

RiskPaths -ini RiskPaths.ini

In Windows you can run the Release version of RiskPaths from inside Visual Studio as follows:

• Solution Configurations to Release and Solution Platforms to x64
• Project Properties > Configuration Properties > Debugging > Command Arguments to
  -ini RiskPaths.ini
• Project Properties > Configuration Properties > Debugging > Working Directory to $(TargetDir)
• To launch the model, do Debug > Start without debugging or Press Ctrl-F5.

When the model run completes, the file RiskPaths.Person.microdata.csv should be present in the model bin directory and look like this:

key,age,union_status,parity_status
1,100,2,1
2,100,2,1
3,100,2,1
4,100,0,1
5,100,0,1

or formatted as a table, like this:

key	age	union_status	parity_status
1	100	2	1
2	100	2	1
3	100	2	1
4	100	0	1
5	100	0	1

The run-time settings output the attributes age, union_status, and parity_status. The leading column key can be used to match microdata records between runs. The build-time option microdata_write_on_exit causes a microdata record to be written whenever an entity leaves the simulation. In RiskPaths there is no mortality and Person entities exit the simulation at age 100. The values of union_status and parity_status are those at that age, for each Person entity in the run.

The model log contains the following warning, which is expected.

Warning : model can expose microdata at run-time with output_microdata = on

[back to quick start]
[back to topic contents]

Worked example 1

This section continues the quick start example to output multiple microdata records for a single entity, at each event.

In ompp_framework.ompp, change the build-time microdata settings to

options microdata_write_on_enter = on;
options microdata_write_on_exit = on;
options microdata_write_on_event = on;

Change the run-time settings in RiskPaths.ini to

[Parameter]
SimulationCases = 1

[Microdata]
ToCsv = yes
Person = age, union_status, parity_status

and run the model.

Here's the resulting microdata output in RiskPaths.Person.microdata.csv, with some repetitive rows elided.

key	age	union_status	parity_status
1	0	0	0
1	1	0	0
1	2	0	0
1	3	0	0
...	...	...	...
1	15	0	0
1	16	0	0
1	17	0	0
1	17.5	0	0
1	18	0	0
1	19	0	0
1	20	0	0
1	21	0	0
1	22	0	0
1	22.5	0	0
1	23	0	0
1	24	0	0
1	24.2609992115357	1	0
1	25	1	0
1	25.2609992115357	1	0
1	26	1	0
1	26.5378127283906	1	1
1	26.5378127283906	1	1
1	27	1	1
1	27.2609992115357	1	1
1	27.2609992115357	2	1
1	27.5	2	1
1	28	2	1
1	29	2	1
1	29.2609992115357	2	1
1	30	2	1
1	31	2	1
1	32	2	1
1	32.5	2	1
1	33	2	1
1	33.2609992115357	2	1
1	34	2	1
1	35	2	1
1	36	2	1
1	37	2	1
1	37.2609992115357	2	1
1	37.5	2	1
1	38	2	1
1	39	2	1
1	40	2	1
1	41	2	1
1	42	2	1
...	...	...	...
1	97	2	1
1	98	2	1
1	99	2	1
1	100	2	1
1	100	2	1
1	100	2	1

The microdata output shows the values of the attributes at every event in the lifecycle. Note that multiple records at the same age are present due to multiple tied events at that age.

Under construction Microdata output can be restricted at run-time to specific named events:

[Microdata]
ToCsv = yes
Person = age, union_status, parity_status
Events = FirstPregEvent, DeathEvent

[back to topic contents]

Worked example 2

This example illustrates how to output microdata whenever a specific event occurs. Specifically, microdata will be output whenever the FirstPregEvent occurs in RiskPaths.

In RiskPaths, prepare the event implement function for hooks by adding the required statement at the end of the event implement function FirstPregEvent:

void Person::FirstPregEvent()
{
    parity_status = PS_PREGNANT;
    hook_FirstPregEvent();
}

Next, add code to hook the built-in function write_microdata to FirstPregEvent:

actor Person {
    hook write_microdata, FirstPregEvent;
};

In ompp_framework.ompp, turn off options which automatically write microdata, which were previously turned on in example 1.

//options microdata_write_on_enter = on;
//options microdata_write_on_exit = on;
//options microdata_write_on_event = on;

The statements inserted in example 1 were commented to revert to the default value off. This means that only explicit calls to write_microdata will generated microdata output.

Set the number of cases to 20 in RiskPaths.ini:

[Parameter]
SimulationCases = 20

[Microdata]
ToCsv = yes
Person = age, union_status, parity_status

Run the model.

The microdata output file RiskPaths.Person.microdata.csv should look like this:

key	age	union_status	parity_status
1	26.5378127283906	1	1
2	24.6780778011483	1	1
3	20.024664717724	1	1
4	17.4107170399441	0	1
5	24.1577392012077	0	1
6	24.7534475294375	1	1
7	18.2797585879836	1	1
8	22.110326319997	1	1
9	21.2430736420085	1	1
10	29.168835553187	1	1
12	37.7955780112222	2	1
14	26.9550960057145	1	1
15	21.6012847802494	0	1
16	20.3178392448776	1	1
18	22.8298415328563	1	1
19	26.7999269606788	1	1
20	19.0257883348614	1	1

The microdata file shows the values of attributes at all occurrences of the FirstPregEvent in the run. It could, for example, be used to chart the distribution of age at first birth using a downstream application like Excel or R.

[back to topic contents]

Worked example 3

This example shows how to produce microdata output using a self-scheduling attribute.

Change the hook in the previous example to

actor Person {
    hook write_microdata, self_scheduling_int(age);
};

and simulate a single case by modifying RiskPaths.ini:

[Parameter]
SimulationCases = 1

[Microdata]
ToCsv = yes
Person = age, union_status, parity_status

Run the model. Microdata output should look like this:

key	age	union_status	parity_status
1	1	0	0
1	2	0	0
1	3	0	0
1	4	0	0
...	...	...	...
1	26	1	0
1	27	1	1
1	28	2	1
1	29	2	1
...	...	...	...
1	100	2	1

The microdata output contains a snapshot of the attributes at each integer age.

[back to topic contents]

Worked example 4

This example outputs microdata containing a population snapshot for the time-based model IDMM.

Under construction

[back to topic contents]

Enabling microdata output

A model is capable of writing microdata if and only if model code contains the following statement:

options microdata_output = on;

A model with microdata capability will write the following warning to the log whenever it is run:

 Warning : model can expose microdata at run-time with microdata_output = on

If this is not a concern, for example if the model generates entities synthetically, this warning can be disabled by the following statement:

options microdata_output_warning = off;

A weight-enabled model which is also microdata-enabled will write the following message to the log when run

Note : model is weight-enabled and microdata-enabled, include entity_weight in Microdata for downstream weighted operations

as a reminder that the attribute entity_weight needs to be included in microdata output for downstream weighted tabulation.

Some internal entity attributes are created by the OpenM++ compiler. For example, the compiler creates an identity attribute to implement the filter of an entity table. These internal entity attributes are normally hidden. They can be made visible, including as microdata, using the following statement:

options all_attributes_visible = on;

[back to topic contents]

Run-time settings

Run time settings are specified as options, on the command line or in an ini file. Below are commented examples of all run-time microdata settings:

[Microdata]

; Person = ageGroup,sex,age,income,isOldAge,pension

; Store all non-internal attributes of Person entity
;
; Person = All

; Store all non-internal attributes of all entities
; NOT recommended for production, use for debug only
;
; All    = true

; Allow to store entities internal attributes
; NOT recommended for production, use for debug only
;
; UseInternal = true

; Write microdata entity attributes into database
; Important: each microdata entity MUST have unique key
;
; ToDb  = false

; Write microdata entity attributes and events (if enabled) into CSV file(s)
; each microdata entity is written in it's own file
;
; ToCsv = false

; Write microdata entity(s) attributes and events (if enabled) into model Trace output
; Trace output must be enabled to produce any results;
; see Trace options in [OpenM] section above
;
; ToTrace = false

; Write selected events into Trace or CSV file
;
; Events = Birth,Union,Death

; If true then write event name into CSV file
;
; CsvEventColumn = true

[back to topic contents]

Build-time settings

Build-time options determine when microdata is written during the simulation of each entity. The available options are:

Option	Default	Description
microdata_write_on_enter	off	microdata is written when the entity enters the simulation, before any event occurs in the entity.
microdata_write_on_exit	off	microdata is written when the entity exits the simulation.
microdata_write_on_event	off	microdata is written after an event occurs in the entity.

These options can be combined. If none of these options are on no microdata will be written unless model code does so explicitly by calling or hooking the built-in function write_microdata.

Note that attributes of an entity can change due to events in other linked entities in a model with interacting entities. So, even if these options are on, changes in attributes of an entity can be absent from microdata output for that entity. To see all changes in attributes in a model with interacting entities, consider using Event Trace.

[back to topic contents]

Model code

Microdata can be written by calling the built-in entity member function write_microdata() from model code, either directly or by using a hook statement.

[back to topic contents]

Microdata key

A key is a unique identifier used to match entities or microdata records across runs. It is a 64-bit value of C++ type uint64_t.

The key for an entity is returned by the entity member function get_entity_key. If this function is not defined in model code, the OpenM++ compiler will provide a definition which returns the value of entity_id.

The key for a microdata output record is produced by the entity member function get_microdata_key If this function is not defined in model code, the OpenM++ compiler will provide a definition which returns the value of the function get_entity_key.

The function xz_crc64 is provided to create a 64-bit key using the crc-64 open source checksum (hash) algorithm. The key can be based on a single value, or on multiple values, as illustrated in the following examples.

Here's a generic example using a location in memory, and a size in bytes.

uint64_t key = xz_crc64(ptrToBytes, sizeInBytes, 0); // calculate crc-64 for bytes array

Here's an example using multiple attributes of a Person entity:

uint64_t key = xz_crc64( &((uint8_t *)(Person->age)), sizeof(int), 0); // start new crc-64 key
key = xz_crc64( &((uint8_t *)(Person->income)), sizeof(double), key);  // continue crc-64 key calculation
key = xz_crc64( &((uint8_t *)(Person->isOld)), sizeof(bool), key);     // continue crc-64 key calculation 

This example just illustrates how to create a key. A real example would use attributes which allow matching across runs.

[back to topic contents]