Bugs in Metrics Calculation

[dolphin-zs]

It seems that the current evaluation code (commit 2de75e8) still has some bugs.

The correct code should be

nan_cond = pd.isna(raw_data).any(axis=1)

Besides, the previous evaluation code is also updated but with a different logic

So, which one should we follow as the final evaluation standard?

[ramber1836]

For the first, yes, you're right. Thanks for pointing that out, this will be fixed soon. For the second, the negative 'Patv' values will be filtered out by raw_data['Patv'] < 0, so the zero 'Patv' values should only be taken into account. All The Best, Xinjiang Lu

…

On Fri, May 6, 2022 at 7:19 PM Shun Zheng ***@***.***> wrote: It seems that the current evaluation code (commit 2de75e8 <2de75e8>) still has some bugs. [image: image] <https://user-images.githubusercontent.com/5029717/167121556-73018798-7902-4dc1-a0cc-45be1e4098ae.png> The correct code should be nan_cond = pd.isna(raw_data).any(axis=1)` raw_data['Patv'] <= 0 — Reply to this email directly, view it on GitHub <#36>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEVHY34UJW6Z2GQMXHH2H2TVIT545ANCNFSM5VH3FV3Q> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[ramber1836]

The current version will be adopted. @dolphin-zs

[dolphin-zs]

BTW, there are two metrics.py files, which one should we use as the evaluation standard?

[DrownFish19]

@ramber1836
Could you provide a detail manual for evaluation? According to the current version, I guess we need to write a method named forecast in the predict.py, which reads the csv data as input and outputs the predicted result. However, I'm not sure if this way is completely correct.

[ramber1836]

OK, I will write a manual for the evaluation which mainly focuses on the proper way to implement your code.

[ramber1836]

The current one should be adopted. All The Best, Xinjiang Lu

…

On Fri, May 6, 2022 at 7:59 PM Shun Zheng ***@***.***> wrote: BTW, there are two metrics.py files, which one should we use as the evaluation standard? — Reply to this email directly, view it on GitHub <#36 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEVHY335KT3FVSU4UR4KJJLVIUCSBANCNFSM5VH3FV3Q> . You are receiving this because you commented.Message ID: ***@***.*** com>

[ramber1836]

The condition is dedicated to the rows with missing values (not just missing target values), thus, I prefer to stick to the current version, i.e. "nan_cond = pd.isna(raw_data).any(axis=1)". The performance reported in the newly updated report is about 42.xx. BTW, the performance you reported is promising. That would be great if your model can keep doing well on the upcoming test data. All The Best, Xinjiang Lu

…

On Fri, May 6, 2022 at 8:05 PM Shun Zheng ***@***.***> wrote: Besides, after using the correct code, 'nan_cond = pd.isna(raw_data).any(axis=1)', the evaluation error can be significantly smaller than the initial one. In my case, I can got about 4.xx on the test set (47.xx reported in your specifications). So, I am curious about whether this nan_cond is proper, maybe pd.isna(raw_data['Patv']) is a better choice. Looking forward to your final evaluation metric. — Reply to this email directly, view it on GitHub <#36 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEVHY36XPQKHBQCQF6ELIP3VIUDH7ANCNFSM5VH3FV3Q> . You are receiving this because you commented.Message ID: ***@***.*** com>