Coefficients definition inside the utility function

[ghost]

Hello,

I have a question regarding the way to define coefficients using the add_shared_coefficients and add_coefficients function.

We have defined the ChoiceDataset as follows:

dataset = ChoiceDataset.from_single_long_df(
    df=df,
    items_id_column='item_id',
    choices_id_column='choice_id',
    shared_features_columns=["store", "city"],
    items_features_columns=["category", "material", "brand", "colour", "price"],
    choices_column='choice',
    choice_format="one_zero"
)

The following is the summary:
%=====================================================================%
%%% Summary of the dataset:
%=====================================================================%
Number of items: 343
Number of choices: 10
%=====================================================================%
Shared Features by Choice:
2 shared features
with names: (['store', 'city'],)

Items Features by Choice:
5 items features
with names: (['category', 'material', 'brand', 'colour', 'price'],)
%=====================================================================%
''

And here is the way we add coefficients:

# Instantiation of the model
cmnl = ConditionalLogit(epochs=10, optimizer="lbfgs")

# Intercept 
cmnl.add_coefficients(feature_name="intercept", items_indexes=list(range(1,343))) # The first intercept needs to be zeroed

# Individual coefficients - context-specific coefficients
cmnl.add_coefficients(feature_name="store", coefficient_name="beta_store", items_indexes=list(range(1,343)))
cmnl.add_coefficients(feature_name="city", coefficient_name="beta_city", items_indexes=list(range(1,343)))

# Shared coefficients
cmnl.add_shared_coefficient(feature_name="price", coefficient_name="beta_price", items_indexes=list(range(343)))
cmnl.add_shared_coefficient(feature_name="category", coefficient_name="beta_category", items_indexes=list(range(343)))
cmnl.add_shared_coefficient(feature_name="brand", coefficient_name="beta_brand", items_indexes=list(range(343)))
cmnl.add_shared_coefficient(feature_name="material", coefficient_name="beta_material", items_indexes=list(range(343)))
cmnl.add_shared_coefficient(feature_name="colour", coefficient_name="beta_colour", items_indexes=list(range(343)))

From your experience with discrete choice models, is it a feasible approach? I was also unsure about the item indexes for the coefficients which are applied individually to each item (intercept, beta_store, beta_city). I've seen in the documentation that the first value of the intercept coefficient needs to be zeroed as a 'baseline', therefore the items_indexes starts from (1,343), leaving the first row out of the coefficients computation, but not sure about the other ones.

[VincentAuriau]

Hello,

I would say that I have a very experimental point of view and one way to validate if a model makes sense is to check if the estimates weights make sens ( e.g. usually a price elasticity should be negative) and if the model is better at predicting choices on a test set than some baselines to be defined (for example a uniform distribution over alternatives).
However here are a few things that might help you:

• 343 different items (alternatives) is a lot just make sure that you have enough data for each of them (I suppose you have more than 10 samples ;) )

• It is indeed usual to define the first intercept to be zero. If you write the probability of an alternative with the softmax of the utility, you can see that adding any $x \in \mathbb{R}$ to all the intercepts does not change the probability value. Fixing one intercept makes sure to get the same results if you run the estimation several times - and helps comparing different formulations.

• It looks like most of your features are categorical (except the price I suppose). What I have seen most of the time is to learn one coefficient by category in the utility:

What you will need to do is to represent such categorical features as one hots so that the model learns one weight by category.

• For the store and the city you will learn a lot of different weights. Again you will need enough data. Depending on your data, you may also observe that learning store specificities will "take over" the specificities of the cities.

• Finally depending on what your alternatives are, if their features (brand, colour, material, category) do not change it is redundant with the intercept. Indeed your utility will be for example

If the price changes, then your utility will change, however if your brand / colour does not change, it basically works like an intercept.

Let me know I am not clear or if you have other questions !

[ghost]

Thanks a lot for the explanations, it helps a lot!
Yes, we do have more than 10 samples - we view each "choice" as a "transactional view" in the sense that we have a product chosen in a transaction (choice=1), and then we retrieve the inventory at that specific point in time to get the alternatives (all the other products that were available in that store at the time of the transaction, choice=0). We first plan to try it out with only a subset, as in this example (10 transactions).

I agree that indeed the "city" coefficient might not bring any useful information since the specificities of the stores might reflect how important it's a store compared to another, and then we can derive demographic conclusions based on those.

One more open question that it's still on our minds is:
Having in mind our scope that we want to retrieve the optimal assortment per store, do you think it will be more suitable to:

1. Train a single "general" model with all the store transaction choices included, and then structure the test set somehow that it focus on each store individually (not sure how this can be done, maybe a separate test set for each store)
2. Train separate models per store (get the specific transactions per store, train a separate ConditionalLogit on each store, and then retrieve the optimal assortments)?
   And if we go with option 2, which might be parallelized and less time-consuming, will we still catch the dependencies between stores e.g. how good is a store compared to another?

Thanks again for this discussion! :)

[VincentAuriau]

I would say that both options seem reasonable and that it makes sense to test both approach and compare them on a well chosen train/ test split. Here are a few "ideas" I came up with that might help you make a choice:

• Are all products sold in all stores ? If no a single model will make more sens if you want to potentially add items in your assortment from other stores
• What formulation makes the most sense ? Having one weight by store for all features & intercepts is the same as having several models. In this case, it will be faster to split the trainings