CGI -- problem with custom filtering

[andreybavt]

GSK-348

When selecting the custom filter, and setting:

• Actual labels = 0
• Predicted labels = 1

one of the data entries shown has:

• Actual labels = 0
• Predicted labels = 0

Upon investigation, when we click on Model --> Inspect (i.e. the inspection is done for the whole dataset) this is the data entry in question:

,ETAB_ACT_DATE_DEBUT_ACTIVITE,ETAB_ACT_MODALITE_EXERCICE,GEST_MAJ_DATE,GDP_FLAG_RGE,ENT_ACT_DATE_IMMAT_RM,ENT_ACT_NON_SEDENTAIRE,ENT_GEST_MAJ_DATE,ENT_GEST_NON_DIFFUSION,DIR_ID_NATIONALITE,DIR_ID_DATE_NAISSANCE,DIR_ID_LISTE_DIFFUSION,DIR_ID_DATE_DEBUT_ACTIVITE,DIR_ADR_PAYS,DIR_FOR_FORAIN,DIR_QA_QUALIFICATION,DIR_QA_DATE,DIR_GEST_MAJ_DATE,INTITULE_ORIGINE_ETABLISSEMENT,INTITULE_CATEGORIE_ETABLISSEMENT,ETAB_NON_SEDENTAIRE,ETAB_AUDIT,DIR_CC_PRESENT,DIR_NATION,TYPOLOGIE_URBAIN_RURAL,ENT_ACT_FORME_JURIDIQUE_2
54,13.971252566735114,Permanente,7.468856947296373,0,13.943874058863791,0,7.468856947296373,False,050,70.48596851471595,0,13.971252566735114,FRANCE,0,SANS QUALIFICATION,121.84804928131418,7.468856947296373,Création,Siège et établissement principal,0,1,0,Fr,urbain densité intermédiaire,54,0

and the prediction (for labels [0, 1]) is:

0.49382649000731726,0.5061735099926827

-------------

Now when skimming through the data, i.e. running prediction only on this entry, this is the resulting prediction (for labels [0, 1]) instead is:

0.72773,0.27227

so completely different predictions.

Project: Giskard
Reporter: rabah
Created: 2022-11-09

[andreybavt]

@rabah-khalek, could you remind me of the status of this issue? I remember we discussed it

[rabah-khalek]

We found out that CGI's model has different results based on the size of the dataset passed to the model. In the last dataset subset, they sent us, one would get different results for:

print(prediction_function(RNM)[54])
print(prediction_function(RNM.iloc[[54]]))

which might hint that they have the equivalent of fit_transform in their prediction_function. We indicated this issue to them, but @jmsquare might have a better idea on the follow-up.

[andreybavt]

Thanks! Seems like nothing else to do on this topic on our side then

[rabah-khalek]

I believe so, this issue can be closed for now.