Docker implementation, pyproject.toml and frozen environment

[yamilbknsu]

Implements Docker container pipeline, documentation, fix random issues.

Things we need to test before merging:

• The environment from pyproject.toml builds correctly (ideally in several platforms, I tested MacOS)
• The instructions to use the .lock file are accurate
• Default configuration file works
• Are we serving the docker image in a dedicated registry?
• Docker compose (and the docker image on its own) run as expected
• Documentation builds and is published correctly

Also any comments regarding the contents (or any other aspects) of the documentation would be great.

closes #68, #19, #4, #60, #33, #18, #17,

[jhoshiko]

After testing, I ran into few issues running the docker-compose up initially because docker attempts to use a local image named image: demos:0.0.1 and when it doesn't find one, it tries to pull the image of the same name from docker hub. This leads to a access denied error. I was able to get around this by first building a demos:0.0.1 local image via docker build -t demos:0.0.1 . and then running docker-compose up. I would propose modifying the docker compose file to include build instructions in the event the user does not have a local copy of the image.

While running in the container, everything seems to work up until the mortality step, where I then encounter a KeyError and a crash. I've attached the runtime log below:

Click to expand

[+] Running 2/2
 ✔ Network demos_nrel_default    Created                                                                                                                                  0.1s 
 ✔ Container demos_nrel-demos-1  Created                                                                                                                                  0.2s 
Attaching to demos-1
demos-1  | 2025-09-04 19:39:35.420 | INFO     | datasources:load_into_orca:24 - Loading CSV 'observed_fatalities_data' table from ../data/calibrated_values/calibration/06197001/mortalities_over_time_obs.csv
demos-1  | 2025-09-04 19:39:35.430 | INFO     | datasources:load_into_orca:24 - Loading CSV 'observed_births_data' table from ../data/calibrated_values/calibration/06197001/births_over_time_obs.csv                                                                                                                                                         
demos-1  | 2025-09-04 19:39:35.442 | INFO     | config:model_post_init:119 - Output file set to default: ../data/output/demos_output_2019.h5
demos-1  | 2025-09-04 19:39:35.443 | INFO     | datasources:load_into_orca:43 - Loading HDF5 'persons' table from ../data/custom_mpo_06197001_model_data.h5/persons
demos-1  | 2025-09-04 19:39:51.946 | INFO     | datasources:load_into_orca:43 - Loading HDF5 'households' table from ../data/custom_mpo_06197001_model_data.h5/households
demos-1  | 2025-09-04 19:40:03.449 | INFO     | datasources:load_into_orca:24 - Loading CSV 'relational_adjustment_mapping' table from ../data/relmap_06197001.csv
demos-1  | 2025-09-04 19:40:03.460 | INFO     | datasources:load_into_orca:24 - Loading CSV 'income_rates' table from ../data/income_rates_06197001.csv
demos-1  | 2025-09-04 19:40:03.465 | INFO     | datasources:load_into_orca:24 - Loading CSV 'hsize_ct' table from ../data/hsize_ct_06197001.csv                                
demos-1  | 2025-09-04 19:40:03.473 | INFO     | datasources:load_into_orca:24 - Loading CSV 'observed_employment' table from ../data/calibrated_values/calibration/06197001/employment_obs.csv
demos-1  | 2025-09-04 19:40:03.481 | INFO     | datasources:load_into_orca:24 - Loading CSV 'observed_marrital_data' table from ../data/calibrated_values/calibration/06197001/marrital_status_over_time_obs.csv                                                                                                                                              
demos-1  | 2025-09-04 19:40:03.669 | INFO     | templates.modelmanager:register:174 - Registering model step 'birth'
demos-1  | 2025-09-04 19:40:03.670 | INFO     | templates.modelmanager:register:174 - Registering model step 'cohabitation'
demos-1  | 2025-09-04 19:40:03.671 | INFO     | templates.modelmanager:register:174 - Registering model step 'enter_labor_force'                                               
demos-1  | 2025-09-04 19:40:03.671 | INFO     | templates.modelmanager:register:174 - Registering model step 'exit_labor_force'                                                
demos-1  | 2025-09-04 19:40:03.671 | INFO     | templates.modelmanager:register:174 - Registering model step 'divorce'
demos-1  | 2025-09-04 19:40:03.671 | INFO     | templates.modelmanager:register:174 - Registering model step 'education'                                                       
demos-1  | 2025-09-04 19:40:03.671 | INFO     | templates.modelmanager:register:174 - Registering model step 'kids_move'                                                       
demos-1  | 2025-09-04 19:40:03.672 | INFO     | templates.modelmanager:register:174 - Registering model step 'marriage'                                                        
demos-1  | 2025-09-04 19:40:03.672 | INFO     | templates.modelmanager:register:174 - Registering model step 'mortality'                                                       
demos-1  | 2025-09-04 19:40:03.674 | INFO     | orca.orca:run:2171 - [external/orca] Running step 'validate_persons_table'                                                     
demos-1  | 2025-09-04 19:40:05.659 | WARNING  | models.data_fix:validate_persons_table:29 - 165735 households with no head. 163055 were single-person households, the person was made head. The other 2680 were dropped.
demos-1  | 2025-09-04 19:40:09.351 | WARNING  | models.data_fix:validate_persons_table:63 - 9 households with multiple partners (53 total people) were dropped
demos-1  | 2025-09-04 19:40:13.496 | WARNING  | models.data_fix:validate_persons_table:93 - 58360 heads have MAR == 1 but no spouse in house. Changing to MAR = 0
demos-1  | 2025-09-04 19:40:13.579 | WARNING  | models.data_fix:validate_persons_table:106 - 6 households have married people but the number of MAR == 1 is different than 2. Spouses were flagged with MAR = 1
demos-1  | 2025-09-04 19:40:19.799 | INFO     | orca.orca:run:2178 - [external/orca] Time to execute step 'validate_persons_table': 16.12 s
demos-1  | 2025-09-04 19:40:19.799 | INFO     | orca.orca:run:2182 - [external/orca] Total time to execute iteration 1 with iteration value None: 16.12 s
demos-1  | 2025-09-04 19:40:19.838 | INFO     | orca.orca:run:2162 - [external/orca] Running iteration 1 with iteration value 2011                                             
demos-1  | 2025-09-04 19:40:19.838 | INFO     | orca.orca:run:2171 - [external/orca] Running step 'aging'
demos-1  | 2025-09-04 19:40:19.878 | INFO     | orca.orca:run:2178 - [external/orca] Time to execute step 'aging': 0.04 s
demos-1  | 2025-09-04 19:40:19.878 | INFO     | orca.orca:run:2171 - [external/orca] Running step 'employment'
demos-1  | 2025-09-04 19:40:25.282 | INFO     | models.employment:run_simultaenous_calibration:220 - Simultaneous Calibration: Iteration 0 error: 102657                       
demos-1  | 2025-09-04 19:40:27.462 | INFO     | models.employment:run_simultaenous_calibration:220 - Simultaneous Calibration: Iteration 1 error: 86882
demos-1  | 2025-09-04 19:40:29.571 | INFO     | models.employment:run_simultaenous_calibration:247 - [external/orca] Final error after Simultaneous calibration: 71274
demos-1  | 2025-09-04 19:40:48.557 | INFO     | orca.orca:run:2178 - [external/orca] Time to execute step 'employment': 28.68 s
demos-1  | 2025-09-04 19:40:48.558 | INFO     | orca.orca:run:2171 - [external/orca] Running step 'household_reorg'
demos-1  | 2025-09-04 19:41:25.606 | DEBUG    | models.household_reorg:simultaneous_calibration:369 - Predicted Marital status after applying models BEFORE CALIBRATION        
demos-1  | 2025-09-04 19:41:25.606 | DEBUG    | models.household_reorg:simultaneous_calibration:370 - Predicted MAR == 1: 2,815,134
demos-1  | 2025-09-04 19:41:25.606 | DEBUG    | models.household_reorg:simultaneous_calibration:371 - Predicted MAR == 3: [554,593, 558,369]
demos-1  | 2025-09-04 19:41:25.607 | INFO     | models.household_reorg:simultaneous_calibration:400 - Simultaneous Calibration: Iteration 0 error: 49987.22302748974           
demos-1  | 2025-09-04 19:41:30.573 | INFO     | models.household_reorg:simultaneous_calibration:400 - Simultaneous Calibration: Iteration 1 error: 48743.85409977344           
demos-1  | 2025-09-04 19:41:35.462 | INFO     | models.household_reorg:simultaneous_calibration:470 - Final error after Simultaneous calibration: 46750.44053268375
demos-1  | 2025-09-04 19:41:38.022 | DEBUG    | models.household_reorg:print_marital_count:514 - Number of people with MAR=1: 2,783,998
demos-1  | 2025-09-04 19:41:38.043 | DEBUG    | models.household_reorg:print_marital_count:514 - Number of people with MAR=3: 521,343
demos-1  | 2025-09-04 19:41:40.697 | DEBUG    | models.household_reorg:household_reorg:120 - Predicted Marital status after applying models
demos-1  | 2025-09-04 19:41:40.697 | DEBUG    | models.household_reorg:household_reorg:121 - Predicted MAR == 1: 2,818,876
demos-1  | 2025-09-04 19:41:40.698 | DEBUG    | models.household_reorg:household_reorg:122 - Predicted MAR == 3: [554,499, 558,705]                                            
demos-1  | 2025-09-04 19:41:40.698 | INFO     | models.household_reorg:household_reorg:125 - Restructuring households:
demos-1  | 2025-09-04 19:41:40.698 | INFO     | models.household_reorg:household_reorg:126 - Cohabitations..                                                                   
demos-1  | 2025-09-04 19:41:47.852 | DEBUG    | models.household_reorg:print_household_stats:622 - Households size from persons table: 2699752
demos-1  | 2025-09-04 19:41:47.853 | DEBUG    | models.household_reorg:print_household_stats:625 - Households size from households table: 2699752
demos-1  | 2025-09-04 19:41:48.679 | DEBUG    | models.household_reorg:print_household_stats:628 - Persons Size: 6758653                                                       
demos-1  | 2025-09-04 19:41:50.693 | DEBUG    | models.household_reorg:print_household_stats:631 - Missing hh: 0
demos-1  | 2025-09-04 19:41:52.446 | DEBUG    | models.household_reorg:print_household_stats:643 - Households with multiple 0: 0
demos-1  | 2025-09-04 19:41:52.450 | DEBUG    | models.household_reorg:print_household_stats:646 - Households with multiple 1: 0
demos-1  | 2025-09-04 19:41:52.455 | DEBUG    | models.household_reorg:print_household_stats:649 - Households with multiple 13: 0                                              
demos-1  | 2025-09-04 19:41:52.462 | DEBUG    | models.household_reorg:print_household_stats:652 - Households with 1 and 13: 0                                                 
demos-1  | 2025-09-04 19:41:52.466 | INFO     | models.household_reorg:household_reorg:132 - Marriages..
demos-1  | 2025-09-04 19:41:59.121 | DEBUG    | models.household_reorg:print_household_stats:622 - Households size from persons table: 2703494                                 
demos-1  | 2025-09-04 19:41:59.122 | DEBUG    | models.household_reorg:print_household_stats:625 - Households size from households table: 2711346
demos-1  | 2025-09-04 19:41:59.122 | DEBUG    | models.household_reorg:print_household_stats:628 - Persons Size: 6758653                                                       
demos-1  | 2025-09-04 19:42:01.002 | DEBUG    | models.household_reorg:print_household_stats:631 - Missing hh: 0                                                               
demos-1  | 2025-09-04 19:42:02.877 | DEBUG    | models.household_reorg:print_household_stats:643 - Households with multiple 0: 0
demos-1  | 2025-09-04 19:42:02.882 | DEBUG    | models.household_reorg:print_household_stats:646 - Households with multiple 1: 0
demos-1  | 2025-09-04 19:42:02.887 | DEBUG    | models.household_reorg:print_household_stats:649 - Households with multiple 13: 0                                              
demos-1  | 2025-09-04 19:42:02.894 | DEBUG    | models.household_reorg:print_household_stats:652 - Households with 1 and 13: 0
demos-1  | 2025-09-04 19:42:04.238 | DEBUG    | models.household_reorg:print_household_stats:622 - Households size from persons table: 2703494                                 
demos-1  | 2025-09-04 19:42:04.238 | DEBUG    | models.household_reorg:print_household_stats:625 - Households size from households table: 2711346
demos-1  | 2025-09-04 19:42:04.238 | DEBUG    | models.household_reorg:print_household_stats:628 - Persons Size: 6758653
demos-1  | 2025-09-04 19:42:06.399 | DEBUG    | models.household_reorg:print_household_stats:631 - Missing hh: 0                                                               
demos-1  | 2025-09-04 19:42:08.122 | DEBUG    | models.household_reorg:print_household_stats:643 - Households with multiple 0: 0
demos-1  | 2025-09-04 19:42:08.126 | DEBUG    | models.household_reorg:print_household_stats:646 - Households with multiple 1: 0
demos-1  | 2025-09-04 19:42:08.130 | DEBUG    | models.household_reorg:print_household_stats:649 - Households with multiple 13: 0                                              
demos-1  | 2025-09-04 19:42:08.137 | DEBUG    | models.household_reorg:print_household_stats:652 - Households with 1 and 13: 0
demos-1  | 2025-09-04 19:42:08.138 | INFO     | models.household_reorg:household_reorg:140 - [external/orca] Divorces..                                                        
demos-1  | 2025-09-04 19:42:13.516 | DEBUG    | models.household_reorg:print_household_stats:622 - Households size from persons table: 2731533                                 
demos-1  | 2025-09-04 19:42:13.516 | DEBUG    | models.household_reorg:print_household_stats:625 - Households size from households table: 2739385
demos-1  | 2025-09-04 19:42:13.516 | DEBUG    | models.household_reorg:print_household_stats:628 - Persons Size: 6758653                                                       
demos-1  | 2025-09-04 19:42:15.557 | DEBUG    | models.household_reorg:print_household_stats:631 - Missing hh: 0
demos-1  | 2025-09-04 19:42:17.553 | DEBUG    | models.household_reorg:print_household_stats:643 - Households with multiple 0: 0
demos-1  | 2025-09-04 19:42:17.557 | DEBUG    | models.household_reorg:print_household_stats:646 - Households with multiple 1: 0
demos-1  | 2025-09-04 19:42:17.562 | DEBUG    | models.household_reorg:print_household_stats:649 - Households with multiple 13: 0
demos-1  | 2025-09-04 19:42:17.569 | DEBUG    | models.household_reorg:print_household_stats:652 - Households with 1 and 13: 0                                                 
demos-1  | 2025-09-04 19:42:22.054 | DEBUG    | models.household_reorg:print_marital_count:514 - Number of people with MAR=1: 2,818,876
demos-1  | 2025-09-04 19:42:22.066 | DEBUG    | models.household_reorg:print_marital_count:514 - Number of people with MAR=3: 557,571
demos-1  | 2025-09-04 19:42:22.091 | INFO     | orca.orca:run:2178 - [external/orca] Time to execute step 'household_reorg': 93.53 s                                           
demos-1  | 2025-09-04 19:42:22.092 | INFO     | orca.orca:run:2171 - [external/orca] Running step 'kids_moving'
demos-1  | 2025-09-04 19:42:41.464 | INFO     | orca.orca:run:2178 - [external/orca] Time to execute step 'kids_moving': 19.37 s                                               
demos-1  | 2025-09-04 19:42:41.464 | INFO     | orca.orca:run:2171 - [external/orca] Running step 'mortality'
demos-1  | Traceback (most recent call last):                                                                                                                                  
demos-1  |   File "/opt/conda/envs/demos-env/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
demos-1  |     return self._engine.get_loc(casted_key)                                                                                                                         
demos-1  |   File "pandas/_libs/index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
demos-1  |   File "pandas/_libs/index.pyx", line 196, in pandas._libs.index.IndexEngine.get_loc                                                                                
demos-1  |   File "pandas/_libs/hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item
demos-1  |   File "pandas/_libs/hashtable_class_helper.pxi", line 7096, in pandas._libs.hashtable.PyObjectHashTable.get_item                                                   
demos-1  | KeyError: 'dead'
demos-1  |                                                                                                                                                                     
demos-1  | The above exception was the direct cause of the following exception:
demos-1  |                                                                                                                                                                     
demos-1  | Traceback (most recent call last):
demos-1  |   File "/demos/simulate.py", line 62, in <module>                                                                                                                   
demos-1  |     run()                                                                                                                                                           
demos-1  |   File "/demos/simulate.py", line 39, in run
demos-1  |     orca.run(                                                                                                                                                       
demos-1  |   File "/opt/conda/envs/demos-env/lib/python3.10/site-packages/orca/orca.py", line 2177, in run                                                                     
demos-1  |     step()
demos-1  |   File "/opt/conda/envs/demos-env/lib/python3.10/site-packages/orca/orca.py", line 973, in __call__                                                                 
demos-1  |     return self._func(**kwargs)
demos-1  |   File "/demos/templates/modelmanager.py", line 180, in run_step                                                                                                    
demos-1  |     return step.run()                                                                                                                                               
demos-1  |   File "/demos/templates/estimated_models/binary_logit.py", line 287, in run
demos-1  |     orca.get_table(tabname).update_col_from_series(colname, df[colname], cast=True)                                                                                 
demos-1  |   File "/opt/conda/envs/demos-env/lib/python3.10/site-packages/orca/orca.py", line 518, in update_col_from_series                                                   
demos-1  |     col_dtype = self.local[column_name].dtype
demos-1  |   File "/opt/conda/envs/demos-env/lib/python3.10/site-packages/pandas/core/frame.py", line 4107, in __getitem__                                                     
demos-1  |     indexer = self.columns.get_loc(key)
demos-1  |   File "/opt/conda/envs/demos-env/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3819, in get_loc                                                  
demos-1  |     raise KeyError(key) from err                                                                                                                                    
demos-1  | KeyError: 'dead'
demos-1  | ERROR conda.cli.main_run:execute(125): `conda run python -u simulate.py -cfg config.toml` failed. (See above for error)                                             
demos-1 exited with code 1