SRBench Competition Discussion Thread

[lacava]

Hi All,

I'm starting this thread to cover general discussion of the SRBench Competition.
Feel free to post questions or comments pertaining to the competition.

About

This competition is building off of the SRBench work but is geared towards unseen datasets. The theme of the competition is "Interpretable Symbolic Regression for Data Science". In that light, we will be testing submissions on both synthetic data challenges as well important, real-world modeling tasks.

A total of $2500 in cash prizes will be split among the winners. Winners will be announced at the Genetic and Evolutionary Computation Conference (GECCO) in Boston, MA this July. Winners will also be awarded an invited speaker slot at the Symbolic Regression Workshop during the conference.

Organizers: @cavalab/srbench-comp

• @lacava, @folivetti, @mkommend , and @marcovirgolin

Sponsors:

• CHIP @ Boston Children's Hospital / Harvard Medical school
• HEAL @ University of Applied Sciences Upper Austria

Important Dates (Deadline Extended to May 15, 2022)

Submissions open: now
Deadline to submit: May 1, 2022 May 15, 2022

How to Submit

The submissions are officially open! Check out the Competition Guide for instructions on how to submit, including an example. Submissions go to the Competition 2022 branch.

[lacava]

Hi All,

After some feedback, we've made some updates/clarification to the Competition Guide:

1. all method runs will fall under the same time guidelines:

• For datasets up to 1000 rows, 60 minutes (1 hour)
• For datasets up to 10000 rows, 600 minutes (10 hours)

Before we had an inconsistency with the PMLB-20 filtering stage that made it difficult to specify running time uniformly.

2. Signal handling: we've added some clarification to how the timeout exception will be handled, including example code.

3. Hyperparameter tuning: we've expanded the examples to show how to handle time limits when you do hyperparameter tuning.

4. CPU cores: we've clarified how to determine what is available on the cluster during a run. The number of CPU cores can be determined via the OMP_NUM_THREADS environmental variable.

As always if you have additional questions feel free to post them here or via email.

[pakamienny]

Hello,
Nice work!
Is there a command we can run locally to make sure the code runs from submission folder?

Otherwise, how is it checked? do you have to manually be there to run it?

Thanks!
Pierre-alexandre

[lacava]

Hi @pakamienny ,

Good question. To start, the submission tests are here: https://github.com/cavalab/srbench/blob/Competition2022/experiment/test_submission.py
and are run via this CI script: https://github.com/cavalab/srbench/blob/Competition2022/.github/workflows/ci.yml

We could add a "local" version of this. Once the method is installed and its environment activated, a test would roughly consist of these commands:

SUBNAME=your-submission-name # submission name
SUBFOLDER=submission/$SUBNAME # path containing submission
SUBENV=srcomp-$SUBNAME # the environment created for your method

# copy it over to the experiment folder
mkdir -p experiment/methods/$SUBNAME
cp $SUBFOLDER/regressor.py experiment/methods/$SUBNAME/
cp $SUBFOLDER/metadata.yml experiment/methods/$SUBNAME/
touch experiment/methods/$SUBNAME/__init__.py

# run the test
cd experiment
mamba run -n $SUBENV python -m pytest -v test_submission.py --ml $SUBNAME         

[foolnotion]

Hi @lacava, does the competition make any assumptions about reproducibility? That is, does the code expect the estimator to return the same result with the same fixed random_state?

[lacava]

we aren't explicitly requiring reproducibility, but would expect submissions to strive for it. we plan to test several realizations of each experiment with different random seeds.