Treat position bias via GAM in LambdaMART #5929

metpavel · 2023-06-15T05:58:00Z

Add support to counter position bias present in relevance labels for learning-to-rank task. Position bias is modeled as an additive factor in logistic space within Generalized Additive Model (GAM) framework.

metpavel · 2023-06-16T04:45:34Z

@microsoft-github-policy-service agree company="Microsoft"

shiyu1994

Left a few comments.

include/LightGBM/dataset.h

src/io/metadata.cpp

include/LightGBM/dataset.h

src/objective/rank_objective.hpp

shiyu1994 · 2023-06-16T09:21:14Z

The static check CI job is failing. And there are 5 linter errors to be fixed from the log. Thanks.
https://github.com/microsoft/LightGBM/actions/runs/5275652092/jobs/9549918023?pr=5929

Linting C++ code
running cpplint
include/LightGBM/dataset.h:224:  An else should appear on the same line as the preceding }  [whitespace/newline] [4]
include/LightGBM/dataset.h:224:  If an else has a brace on one side, it should have it on both  [readability/braces] [5]
include/LightGBM/dataset.h:237:  An else should appear on the same line as the preceding }  [whitespace/newline] [4]
include/LightGBM/dataset.h:237:  If an else has a brace on one side, it should have it on both  [readability/braces] [5]
src/io/metadata.cpp:576:  Line ends in whitespace.  Consider deleting these extra spaces.  [whitespace/end_of_line] [4]

src/io/metadata.cpp

src/objective/rank_objective.hpp

shiyu1994

Two minor revisions.

jameslamb · 2023-06-19T06:17:53Z

/gha run r-valgrind

Workflow R valgrind tests has been triggered! 🚀
https://github.com/microsoft/LightGBM/actions/runs/5308500765

Status: success ✔️.

tests/python_package_test/test_engine.py

jameslamb

Alright I've quickly skimmed the papers linked the documentation, and think I finally understand now how this is working.

I'm ok with removing the parameter to provide a customized filename... thanks @shiyu1994 for the explanation in #5929 (comment).

I'm comfortable with the changes to LightGBM's Python and C API (other than a minor comment about parameter naming) and I'd be happy to implement the R side of this after this PR is merged. Can one of you please create a new issue at https://github.com/microsoft/LightGBM/issues to track that work?

Please see my most recent round of comments. I still feel that:

the tests as currently written don't provide very strong evidence of this implementation's correctness
the documentation is unclear about the expected content of positions

I'll watch this PR closely this week and try to respond quickly as you comment and ask for new reviews. I understand there's some urgency inside Microsoft on getting this merged. But please note that I'll be traveling Tuesday-Thursday with limited available time here on GitHub.

docs/Parameters.rst

src/io/metadata.cpp

tests/python_package_test/test_engine.py

docs/Advanced-Topics.rst

jameslamb · 2023-08-14T14:37:58Z

/gha run r-valgrind

Workflow R valgrind tests has been triggered! 🚀
https://github.com/microsoft/LightGBM/actions/runs/5857085872

Status: success ✔️.

Co-authored-by: James Lamb <jaylamb20@gmail.com>

metpavel · 2023-08-26T09:18:04Z

@jameslamb, thank you for your effort to learn more about the problem and the proposed solution! And thank you for offering your help in adding the R language support for it. Here is the issue I created for that: #6063.

jameslamb

Thanks so much for accepting my proposed rewording for the documentation, for the effort you put into the excellent new tests, and for documenting the R addition in #6063.

Please fix the compilation errors (looks like there are a few more places where lambdarank_position_bias_regularizer needs to be changed to lambdarank_position_bias_regularization). After that, I'll run the valgrind checks one more time.

Besides that, I have no other major comments left that are worth blocking merging. There are some minor code things I'd like to change in the tests, but I'll propose that in a follow-up PR after this.

Now that I understand it (thanks for your patience with me 😅 ) I'm very excited to see this feature make it into LightGBM! It's been 2.5 years since @robhowley first introduced #4531. Thanks to you and @shiyu1994 for all your efforts in making this happen.

I also want to tag @thvasilo, who was very involved in the discussion in #4531, just so they're aware.

docs/Advanced-Topics.rst

src/io/metadata.cpp

jameslamb · 2023-09-03T05:54:10Z

tests/python_package_test/test_engine.py

+    gbm_unbiased_with_file = lgb.train(params, lgb_train, valid_sets = lgb_valid, num_boost_round=50)
+
+    # the performance of the unbiased LambdaMART should outperform the plain LambdaMART on the dataset with position bias
+    assert gbm_baseline.best_score['valid_0']['ndcg@3'] + 0.03 <= gbm_unbiased_with_file.best_score['valid_0']['ndcg@3']


Excellent, thank you for this! I think we'll be glad to have this stricter test.

tests/python_package_test/test_engine.py

Co-authored-by: James Lamb <jaylamb20@gmail.com>

jameslamb · 2023-09-04T03:21:19Z

/gha run r-valgrind

Workflow R valgrind tests has been triggered! 🚀
https://github.com/microsoft/LightGBM/actions/runs/6068729923

Status: success ✔️.

jameslamb

All of my remaining comments have been addressed by @shiyu1994 's recent comments, so marking this "approved".

Please do wait to see if the valgrind test passes one more time before merging though: #5929 (comment)

Thank you both for all the hard work that went into this addition to LightGBM!

…ghtGBM into HEAD

metpavel · 2023-09-06T05:33:40Z

All of my remaining comments have been addressed by @shiyu1994 's recent comments, so marking this "approved".
Thank you both for all the hard work that went into this addition to LightGBM!

@jameslamb, thank you again for your diligent reviews and prioritizing the quality of LightGBM! Many thanks to @shiyu1994 for your help and support! I'm looking forward to future collaborations with you guys!

github-actions · 2023-12-06T00:20:50Z

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.

metpavel added 3 commits June 14, 2023 21:48

Update dataset.h

4fea7aa

Update metadata.cpp

e1bbbba

Update rank_objective.hpp

6027716

metpavel requested review from guolinke and shiyu1994 as code owners June 15, 2023 05:58

Update metadata.cpp

199d412

jameslamb added in progress feature labels Jun 15, 2023

shiyu1994 reviewed Jun 16, 2023

View reviewed changes

src/objective/rank_objective.hpp Show resolved Hide resolved

metpavel added 3 commits June 16, 2023 21:22

Update rank_objective.hpp

1a56b5a

Update metadata.cpp

93f67d1

Update dataset.h

c615b7d

shiyu1994 reviewed Jun 19, 2023

View reviewed changes

src/io/metadata.cpp Outdated Show resolved Hide resolved

shiyu1994 reviewed Jun 19, 2023

View reviewed changes

src/objective/rank_objective.hpp Outdated Show resolved Hide resolved

shiyu1994 reviewed Jun 19, 2023

View reviewed changes

src/objective/rank_objective.hpp Outdated Show resolved Hide resolved

shiyu1994 reviewed Jun 19, 2023

View reviewed changes

metpavel added 3 commits June 19, 2023 20:10

Update rank_objective.hpp

e374e9a

Update metadata.cpp

6c7b86f

Update test_engine.py

365ca75

metpavel requested review from StrikerRUS, jmoralez and jameslamb as code owners June 20, 2023 03:13

jameslamb requested changes Jun 20, 2023

View reviewed changes

tests/python_package_test/test_engine.py Outdated Show resolved Hide resolved

tests/python_package_test/test_engine.py Outdated Show resolved Hide resolved

metpavel added 3 commits June 19, 2023 20:19

Update test_engine.py

9f033ed

Add files via upload

50659d7

Update test_engine.py

45fbe8b

jameslamb requested changes Aug 14, 2023

View reviewed changes

docs/Parameters.rst Outdated Show resolved Hide resolved

src/io/metadata.cpp Show resolved Hide resolved

tests/python_package_test/test_engine.py Show resolved Hide resolved

docs/Advanced-Topics.rst Outdated Show resolved Hide resolved

metpavel and others added 7 commits August 25, 2023 01:55

Update test_engine.py

5efb9a9

Update test_engine.py

ee2fc4b

Update test_engine.py

98ac42d

Update docs/Advanced-Topics.rst

ca4bd04

Co-authored-by: James Lamb <jaylamb20@gmail.com>

Update Parameters.rst

56a337f

Update rank_objective.hpp

2b88856

Update config.h

893405b

metpavel mentioned this pull request Aug 26, 2023

[R-package] Add R support for Position-Bias-aware Learning-to-Rank #6063

Open

jameslamb self-requested a review September 3, 2023 05:58

jameslamb requested changes Sep 3, 2023

View reviewed changes

shiyu1994 and others added 3 commits September 4, 2023 10:21

Merge branch 'master' into metpavel-posbias_GAM

5dd6428

update config_auto.cppp

f758b26

Update docs/Advanced-Topics.rst

960c758

Co-authored-by: James Lamb <jaylamb20@gmail.com>

jameslamb self-requested a review September 4, 2023 03:24

jameslamb approved these changes Sep 4, 2023

View reviewed changes

shiyu1994 added 2 commits September 4, 2023 06:13

fix randomness in test case for gpu

3d934e6

Merge branch 'metpavel-posbias_GAM' of https://github.com/metpavel/Li…

ad188f6

…ghtGBM into HEAD

shiyu1994 merged commit 7e34d23 into microsoft:master Sep 4, 2023
41 checks passed

jameslamb mentioned this pull request Sep 5, 2023

Release v4.1.0 #6076

Merged

19 tasks

metpavel deleted the metpavel-posbias_GAM branch September 6, 2023 04:33

jameslamb mentioned this pull request Sep 7, 2023

[ci] [docs] fix broken ACM links #6083

Merged

shiyu1994 mentioned this pull request Oct 9, 2023

[Question] Is Microsoft still supporting this project? #6128

Closed

github-actions bot removed the in progress label Dec 6, 2023

github-actions bot locked as resolved and limited conversation to collaborators Dec 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Treat position bias via GAM in LambdaMART #5929

Treat position bias via GAM in LambdaMART #5929

metpavel commented Jun 15, 2023

metpavel commented Jun 16, 2023

shiyu1994 left a comment

shiyu1994 commented Jun 16, 2023

shiyu1994 left a comment

jameslamb commented Jun 19, 2023 •

edited by guolinke

jameslamb left a comment

jameslamb commented Aug 14, 2023 •

edited by guolinke

metpavel commented Aug 26, 2023

jameslamb left a comment

jameslamb Sep 3, 2023

jameslamb commented Sep 4, 2023 •

edited by guolinke

jameslamb left a comment

metpavel commented Sep 6, 2023

github-actions bot commented Dec 6, 2023

Treat position bias via GAM in LambdaMART #5929

Treat position bias via GAM in LambdaMART #5929

Conversation

metpavel commented Jun 15, 2023

metpavel commented Jun 16, 2023

shiyu1994 left a comment

Choose a reason for hiding this comment

shiyu1994 commented Jun 16, 2023

shiyu1994 left a comment

Choose a reason for hiding this comment

jameslamb commented Jun 19, 2023 • edited by guolinke

jameslamb left a comment

Choose a reason for hiding this comment

jameslamb commented Aug 14, 2023 • edited by guolinke

metpavel commented Aug 26, 2023

jameslamb left a comment

Choose a reason for hiding this comment

jameslamb Sep 3, 2023

Choose a reason for hiding this comment

jameslamb commented Sep 4, 2023 • edited by guolinke

jameslamb left a comment

Choose a reason for hiding this comment

metpavel commented Sep 6, 2023

github-actions bot commented Dec 6, 2023

jameslamb commented Jun 19, 2023 •

edited by guolinke

jameslamb commented Aug 14, 2023 •

edited by guolinke

jameslamb commented Sep 4, 2023 •

edited by guolinke