MRA modeling fails when one of the datasets has a constant field

[connorschwartz]

When doing exploratory testing with small subsets of data, I sometimes had situations where one of the columns in the test data was constant, i.e. every row had the same value for that subset of the data.

There is code in the MRA modeling method which adds a constant column to each dataframe, to represent the intercept value in the model:

openavmkit/openavmkit/modeling.py

Line 1657 in 295c080

if intercept:

The add_constant method does not add a new column if there's already a constant column in the dataframe. Because of this, if one of the columns happens to be constant in the X_test dataframe but not in the other dataframes, we end up in a situation where X_test has (n) columns and the other dataframes have (n + 1) columns. The mismatched column count leads to failures in the prediction phase after the fitted model is created.

I think using the has_constant='add' argument when running add_constant would prevent this issue - see https://www.statsmodels.org/stable/generated/statsmodels.tools.tools.add_constant.html

[larsiusprime]

Alright, I'm back from traveling. Will try to reproduce this and deal with it. Might be able to create something right in the test harness.

[larsiusprime]

Hmmm, I might need some help tracking this down. I'm trying to create a minimal test case that I can add to the test suite, and figure out a scenario in which it will fail under current conditions, then fix the code based on that so that the test succeeds.

Here's what I've got so far, I added this locally to test_modeling.py:

def test_mra():
    data = {
        "key": ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19"],
        "key_sale": ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19"],
        "bldg_area_finished_sqft": [
            1000, 1000, 1000, 1000, 
            1500, 1500, 1500, 1500, 
            2000, 2000, 2000, 2000,
            2500, 2500, 2500, 2500,
            3000, 3000, 3000, 3000
        ],
        "land_area_sqft": [
            10000, 10000, 10000, 10000,
            20000, 20000, 20000, 20000,
            30000, 30000, 30000, 30000,
            40000, 40000, 40000, 40000,
            50000, 50000, 50000, 50000
        ],
        "bldg_type_A": [
            1, 1, 1, 1,
            0, 0, 0, 0,
            0, 0, 0, 0,
            1, 1, 1, 1,
            0, 0, 0, 0
        ],
        "bldg_type_B": [
            0, 0, 0, 0,
            1, 1, 1, 1,
            1, 1, 1, 1,
            0, 0, 0, 0,
            1, 1, 1, 1
        ],
        "location_A": [
            1, 1, 1, 1,
            0, 0, 0, 0,
            0, 0, 0, 0,
            1, 1, 1, 1,
            0, 0, 0, 0
        ],
        "location_B": [
            0, 0, 0, 0,
            1, 1, 1, 1,
            1, 1, 1, 1,
            0, 0, 0, 0,
            1, 1, 1, 1
        ],
        "flat": [
            1, 1, 1, 1,
            1, 1, 1, 1,
            1, 1, 1, 1,
            1, 1, 1, 1,
            1, 1, 1, 1
        ],
        "model_group": [
            "a", "a", "a", "a",
            "a", "a", "a", "a",
            "a", "a", "a", "a",
            "a", "a", "a", "a",
            "a", "a", "a", "a"
        ]
    }
    
    df = pd.DataFrame(data)
    df["sale_price"] = df["bldg_area_finished_sqft"] * 20
    df["sale_price"] += df["bldg_type_A"] * df["bldg_area_finished_sqft"] * 2
    df["sale_price"] += df["bldg_type_B"] * df["bldg_area_finished_sqft"] * 5
    df["sale_price"] += df["location_A"] * df["land_area_sqft"] * 1
    df["sale_price"] += df["location_B"] * df["land_area_sqft"] * 2
    df["valid_sale"] = True
    df["vacant_sale"] = False
    df["sale_date"] = "2025-01-01"
    df["sale_date"] = pd.to_datetime(df["sale_date"], format="%Y-%m-%d")
    df["is_vacant"] = False
    df["valid_for_ratio_study"] = True
    df["sale_age_days"] = 0
    
    ind_vars = ["bldg_area_finished_sqft", "land_area_sqft", "bldg_type_A", "bldg_type_B", "location_A", "location_B", "flat"]
    df_sales = df.copy()
    df_universe = df[(["key","is_vacant"]+ind_vars)].copy()
    test_keys = ["0", "1", "2", "3", "4", "5"]
    train_keys = ["6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19"]
    
    ds = DataSplit(
        df_sales,
        df_universe,
        "a",
        {},
        "sale_price",
        "sale_price",
        ind_vars,
        [],
        {},
        test_keys,
        train_keys
    )
    run_mra(ds, intercept=True)

As I understand this bug report, the issue is with the function run_mra(), and it should trigger when intercept=True AND there is already a "constant" variable (ie, a variable where all values are the same, e.g. 1).

I've attempted to do that here with the variable "flat". Can you add this test to your test_modeling.py, run pytest -v -k test_mra and tell me what you get? Right now the test passes for me, whereas I'm expecting it to fail based on your bug report, indicating that either this silently got fixed somehow already, or I'm misunderstanding the reproduction conditions.

[connorschwartz]

Hey Lars - the issue is specific to cases where one of the columns is constant for one part of the data split, but not the other, e.g. if a field is constant for the Test split but not for the Train split. To reproduce the issue with your example, change the "flat" column to this:
"flat": [
1, 1, 1, 1,
1, 1, 1, 1,
1, 1, 1, 1,
1, 1, 1, 1,
1, 1, 1, 0
],

This way, the field will be constant for the Test set but not the Train set. I tried this locally and was able to reproduce the issue.

[larsiusprime]

Nice catch!