scores all zeros despite getting >0.3 on kaggle submission

[yxc8]

Hi,

it seems like I'm getting all zero scores when running pipenv run python -m src.evaluate --test-labels test_labels.jsonl --predictions predictions.csv where the test_labels.jsonl refer to the file generated by pipenv run python -m src.testset --train-set train.jsonl --days 2 --output-path 'out/' --seed 42 . However, submitting the same csv to kaggle gives score >0.3. Any suggestions on what might be causing this?

INFO:root:Reading labels from out/test_labels.jsonl
Preparing labels: 100%|█████████████| 515702/515702 [00:05<00:00, 101878.16it/s]
INFO:root:Read 515702 labels
INFO:root:Reading predictions from predictions.csv
Preparing predictions: 100%|███████| 5015409/5015409 [06:25<00:00, 13000.44it/s]
INFO:root:Read 1671803 predictions
INFO:root:Calculating scores
Evaluating sessions: 100%|██████████| 515702/515702 [00:01<00:00, 324037.41it/s]
INFO:root:Scores: {'clicks': 0.0, 'carts': 0.0, 'orders': 0.0, 'total': 0.0}

Thank you!

[philippnormann]

Hi @yxc8, are you sure you scored the test_sessions.jsonl and not the test.jsonl from Kaggle for your local CV?

[yxc8]

Ah, I see, thanks for pointing this out! I was indeed using the wrong file.

One more thing, the current test still fails because it can't catch the case when prediction doesn't have a certain key. So in evaluate.py, evaluate_session() function, change if 'clicks' in labels and labels['clicks']: to if 'clicks' in labels and labels['clicks'] and 'clicks' in prediction.keys(): and similarly for carts and orders, then the test works without error. Just want to point this out in case someone run into a similar problem.

Anyway thanks for the help Philip!

[philippnormann]

Thanks for pointing that out, @yxc8!