Add PySR & SymbolicRegression.jl to suite #62

MilesCranmer · 2022-01-15T04:05:54Z

This adds the install script and the scikit-learn API. I also added a Julia install step to the GitHub action.

I wasn't sure what the tuned directory was for so I have left it for now. I also wasn't sure how you typically deal with choices of operators so I have added some different choices to the hyperparameter search.

Let me know what else I need to add. Thanks!
Cheers,
Miles

lacava · 2022-01-19T15:09:22Z

Hi @MilesCranmer , thanks for this PR! Looks like there's a TypeError being thrown:

ml = 'PySRRegressor'

    @pytest.mark.parametrize("ml", MLs)
    def test_evaluate_model(ml):
        print('running test_evaluate_model with ml=',ml)
        dataset = 'test/192_vineyard_small.tsv.gz'
        results_path = 'tmp_results'
        random_state = 42
    
        algorithm = importlib.__import__('methods.'+ml,globals(),
                                         locals(),
                                       ['est','hyper_params','complexity'])
    
        print('algorithm imported:',algorithm)
        evaluate_model(dataset,
                       results_path,
                       random_state,
                       ml,
                       algorithm.est,
                       algorithm.hyper_params,
                       algorithm.complexity,
                       algorithm.model,
>                      test=True # testing
                      )

test_evaluate_model.py:38: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
evaluate_model.py:139: in evaluate_model
    print('training',grid_est)
/usr/share/miniconda/envs/srbench/lib/python3.7/site-packages/sklearn/base.py:260: in __repr__
    repr_ = pp.pformat(self)
/usr/share/miniconda/envs/srbench/lib/python3.7/pprint.py:144: in pformat
    self._format(object, sio, 0, 0, {}, 0)
/usr/share/miniconda/envs/srbench/lib/python3.7/pprint.py:161: in _format
    rep = self._repr(object, context, level)
/usr/share/miniconda/envs/srbench/lib/python3.7/pprint.py:393: in _repr
    self._depth, level)
/usr/share/miniconda/envs/srbench/lib/python3.7/site-packages/sklearn/utils/_pprint.py:181: in format
    changed_only=self._changed_only)
/usr/share/miniconda/envs/srbench/lib/python3.7/site-packages/sklearn/utils/_pprint.py:437: in _safe_repr
    v, context, maxlevels, level, changed_only=changed_only)
/usr/share/miniconda/envs/srbench/lib/python3.7/site-packages/sklearn/utils/_pprint.py:446: in _safe_repr
    rep = repr(object)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <[TypeError("'float' object is not subscriptable") raised in repr()] PySRRegressor object at 0x7fcf7094a750>

    def __repr__(self):
>       return f"PySRRegressor(equations={self.get_best()['sympy_format']})"
E       TypeError: 'float' object is not subscriptable

methods/PySRRegressor.py:48: TypeError
----------------------------- Captured stdout call -----------------------------
running test_evaluate_model with ml= PySRRegressor
algorithm imported: <module 'methods.PySRRegressor' from '/home/runner/work/srbench/srbench/experiment/methods/PySRRegressor.py'>
========================================
Evaluating PySRRegressor on 
test/192_vineyard_small.tsv.gz
========================================
compression: gzip
filename: test/192_vineyard_small.tsv.gz
scaling X
scaling y
X_train: (15, 3)
y_train: (15,)
test mode enabled
hyper_params set to {}
training 
=============================== warnings summary ===============================

also, OperonRegressor (@foolnotion ) is failing - is this something new @foolnotion ?

foolnotion · 2022-01-19T15:20:02Z

@lacava yes, there have been some major changes, sorry about that. I will PR an updated install script.

MilesCranmer · 2022-01-20T15:02:38Z

Thanks, just fixed!

lacava · 2022-01-21T14:41:36Z

hm, still seeing a type error...

folivetti · 2022-01-21T14:48:40Z

it seems your PySRRegressor isn't sklearn compatible. Have a look at this code https://gist.github.com/folivetti/609bc9b854c51968ef90aa675ccaa60d

I guess you just need to import from sklearn.base import BaseEstimator, RegressorMixin and then change class PySRRegressor: to class PySRRegressor(BaseEstimator, RegressorMixin):

MilesCranmer · 2022-01-21T19:17:54Z

Fixed!

MilesCranmer · 2022-01-22T22:08:40Z

It should actually work now. Sorry for the delay. The way PySR was being ran inside the srbench test changed the definition of stdin within julia (called with PyJulia) and it caused some type errors. Fixed it now and seems to be running.

MilesCranmer · 2022-01-23T00:52:59Z

Here's it working (using the PR #65 for quicker CI): https://github.com/MilesCranmer/srbench/runs/4909749492?check_suite_focus=true

I reduced the number of possible hyperparams since it was fairly slow. Will the hyperparams I give affect the benchmark, or will you use different ones? The hyperparams I listed were for speedy testing but may not have great performance.

MilesCranmer · 2022-01-25T01:31:12Z

@lacava I've been trying to get julia built directly inside the conda environment but it's much more difficult than I thought. Is it okay if I switch back to building it from the github action as before? It seemed to work with that.

MilesCranmer · 2022-01-25T05:13:56Z

Nevermind, seems like its working directly from conda now! Ready for review.

lacava · 2022-01-25T05:28:22Z

I wasn't sure what the tuned directory was for so I have left it for now. I also wasn't sure how you typically deal with choices of operators so I have added some different choices to the hyperparameter search.

we'll populate the tuned directory for methods that are tested on PMLB. check out issue #24 for a discussion on setting the hyperparameters.

MilesCranmer · 2022-01-25T12:12:51Z

Thanks, will read in-depth soon. Just so I understand, you are limiting methods to 500k evaluations, as in 500k total equations tried on a dataset? PySR can do 500k evaluations in ~1 second on a 4-core laptop... would this mean that PySR's benchmark will only allow for 1 second of search time? (PySR's algorithm is very evaluation-oriented)

Maybe there is a fixed-wall clock time test where PySR can be benchmarked as well?

(The reason it can get to this speed is basically because I optimized the equation evaluation for like six months... I even fuse subtrees of operators into a single compiled kernel to get a speedup! At one point I even tried automatically caching parts of equations too.)

gkronber · 2022-01-25T12:58:08Z

PySR can do 500k evaluations in ~1 second on a 4-core laptop...

This sounds impressive and would be at least an order of magnitude faster than operon.

I assume the number of evaluations per second depends on the number of rows in the dataset and on the avg. size of individuals.
Please correcty me if I'm wrong but assuming 200 rows and an average size of 50 operations for an individual this means that PySR reaches approximately 5e5 * 200 * 50 = 5 GFLOPS on your notebook.

MilesCranmer · 2022-01-25T17:22:35Z

Here's an example with the equation x1*x2 + cos(x3), with 200 rows, using PySR's julia API SymbolicRegression.jl.

using BenchmarkTools
using SymbolicRegression

# 200 rows, 3 features:
X = randn(3, 200)

# Enable + * / - cos sin operators:
options = Options(binary_operators=(+, *, /, -), unary_operators=(cos, sin))

# Create equation
tree = Node("x1") * Node("x2") + cos(Node("x3"))

# Set up evaluation function:
testfunc() = evalTreeArray(tree, X, options)

# Evaluate performance:
@btime testfunc()

This gives 1.817 us on my laptop.

So 4x cores would be equivalent to 2,201,430 evaluations per second. In practice despite the optimization, the equation eval is still the most expensive part so the mutation parts shouldn't add much.

(The equation itself isn't compiled, but the operations used to evaluate the tree are, so this performance be similar for a random tree)

lacava · 2022-01-25T18:57:18Z

This discussion should probably be elsewhere than this PR, but in short, our goal with SRBench has been to benchmark methods, not implementations (to the extent possible). We do measure and compare running time alongside accuracy and complexity and report them all together. FWIW, the synthetic benchmark upped the evals to 1M, but removed the tuning step (hence the `tuned' folder).

MilesCranmer · 2022-01-25T19:12:21Z

I agree this makes sense in principle, but often choices in the implementation inform what methods are even possible, so I don't think these should be disentangled. For example, in PySR there are several choices made in designing the algorithm to increase evaluation speed, which changed the underlying evolutionary algorithm*. Maybe the best compromise would be to have separate benchmarks - one for raw wall clock performance, and one for sample efficiency? From a user's perspective, I think the wall clock benchmark would be much more useful in terms of what tool to use.

But yes, maybe best left to separate thread.

* for example, it's often faster to split a population of individuals into small fixed-size groups, then operate on the groups independently (in parallel) before merging, even though this changes the algorithm itself.

lacava · 2022-01-25T22:12:42Z

the methods all seem to be passing the tests, but then this is happening at the end:

free(): invalid pointer
/home/runner/work/_temp/32bbc723-8bfa-4eae-a161-16dc7a15da38.sh: line 4: 12210 Aborted                 (core dumped) python -m pytest -v test_evaluate_model.py
Error: Process completed with exit code 134.

MilesCranmer · 2022-01-25T23:20:54Z

Hm, this is interesting. So this is obviously a C function, yet PySR has no C dependencies in the main search code which is pure Julia. The other PySR deps are very stable libraries like numpy/sympy which I wouldn't expect to give such an issue.

However, I know that PyJulia directly links into the Python binary which can change some behaviour, like signals.

Does this script run all algorithms in one single kernel evaluation, or does it start a new Python kernel for each algorithm? I wonder if there is some interaction between algorithms.

MilesCranmer · 2022-01-26T03:32:00Z

Edited test_evaluate_model.py to take a single algorithm as an argument. Then, test_evaluate_model.sh to replicate the loop over all *.py files except __init__.py in the methods folder. This will re-launch the python kernel each algorithm, which should prevent interference.

lacava · 2022-01-30T23:37:22Z

I reduced the number of possible hyperparams since it was fairly slow. Will the hyperparams I give affect the benchmark, or will you use different ones? The hyperparams I listed were for speedy testing but may not have great performance.

The hyperparams you give will affect the benchmark (whenever we fix them and run it). For v2.0, we limited to six combos.

However, many hyperparameters shouldn't affect the speed of the tests. When you call evaluate_model.py with the tests=True flag, it skips tuning. You can edit that conditional if you want to further restrict settings for the test. (Probably should handle that better in the future)

lacava · 2022-01-30T23:43:32Z

Does this script run all algorithms in one single kernel evaluation, or does it start a new Python kernel for each algorithm? I wonder if there is some interaction between algorithms.

For tests, pytest is a consistent kernel I imagine. I guess that is why you turned it into a script.
For the experiment, new kernel. we're running batch scripts to evaluate in analyze.py

lacava

looks good, I have one question about Julia in the CI setup. Otherwise ready to merge I think.

lacava · 2022-01-30T23:50:31Z

.github/workflows/test.yml

@@ -49,6 +49,14 @@ jobs:
          conda list
          conda activate srbench

+      # This is useful for up-to-date CI, but


I'm not too familiar with Julia, what is happening here? Is it over an above including Julia in environment.yml?

It it's going to potentially break CI when Julia packages update, i'd rather use a fixed version and update that as needed for srbench dependencies.

This can be removed now. Basically, there is a problem with the Julia package servers in that sometimes they are up to a few days out-of-date. I included the most up-to-date PySR version in this PR which required this forced package server update. However, it's been a few days so the servers are all likely up-to-date!

i sent a PR to your fork for this

.github/workflows/test.yml

remove julia pkg server, add PySR to readme

Add PySR & SymbolicRegression.jl to suite

MilesCranmer mentioned this pull request Jan 15, 2022

add methods #4

Open

9 tasks

MilesCranmer mentioned this pull request Jan 21, 2022

Split GitHub actions into different action for each algorithm #65

Closed

4 tasks

MilesCranmer added 15 commits January 21, 2022 17:29

Create PySR install script

9ee6ff2

Install Julia to GitHub action

dff869f

Add sklearn API for PySR

e6a4200

Return string instead of sympy format

60935f2

Remove list of all PySR settings

f8f9707

Test extra unary operators

437bd4f

Add PySRRegressor to list of algorithms

5d5590b

Fix mistaken PySR call

aa728a3

Fix missing comma

37d66f8

Return 0.0 if no equation currently stored

4a96df3

Fix format of get_best() on unfitted model

fab8da8

Make compatible with sklearn

db06902

Fix get_params missing kwarg

55c1044

Fix nprocs argument in pysr

d3b36dd

Bump pysr version

0052d7f

MilesCranmer force-pushed the master branch from b92c482 to 0052d7f Compare January 21, 2022 22:30

MilesCranmer added 3 commits January 21, 2022 17:30

Reduce Julia version

b8b784d

Use git-based Julia package server

5d8faee

Bump PySR version with stdin fix

fbe422c

MilesCranmer added 2 commits January 22, 2022 17:28

Reduce PySR hyperparams for quick testing

28b4ac3

Fix key error in complexity calculation

9dcc24d

MilesCranmer added 2 commits January 24, 2022 19:27

Add PySR and Julia to conda env

e1b0eaa

Add LLVM and Libm to env

6423b70

MilesCranmer added 3 commits January 24, 2022 21:07

Remove duplicate julia install from CI

b1be6d4

Fix openlibm compatibility

3a33e74

Misplaced = in pip install

69e9b08

Remove duplicate openlibm in environment.yml

f4cad8b

MilesCranmer mentioned this pull request Jan 25, 2022

Wall clock time benchmark #67

Closed

MilesCranmer added 2 commits January 25, 2022 22:30

Re-launch python kernel for each algorithm

f2bb3fb

Update CI to run algorithm launcher script

9ab4015

Turn on test evaluation again

de2a119

lacava reviewed Jan 30, 2022

View reviewed changes

remove julia pkg server, add PySR to readme

8e5ae42

lacava mentioned this pull request Feb 2, 2022

remove julia pkg server, add PySR to readme MilesCranmer/srbench#1

Merged

Merge pull request #1 from cavalab/pr62

a34ec85

remove julia pkg server, add PySR to readme

lacava merged commit dffb18c into cavalab:master Feb 2, 2022

gAldeia pushed a commit to gAldeia/srbench that referenced this pull request May 29, 2024

Merge pull request cavalab#62 from MilesCranmer/master

4095690

Add PySR & SymbolicRegression.jl to suite

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add PySR & SymbolicRegression.jl to suite #62

Add PySR & SymbolicRegression.jl to suite #62

MilesCranmer commented Jan 15, 2022

lacava commented Jan 19, 2022

foolnotion commented Jan 19, 2022

MilesCranmer commented Jan 20, 2022

lacava commented Jan 21, 2022

folivetti commented Jan 21, 2022

MilesCranmer commented Jan 21, 2022

MilesCranmer commented Jan 22, 2022

MilesCranmer commented Jan 23, 2022

MilesCranmer commented Jan 25, 2022

MilesCranmer commented Jan 25, 2022

lacava commented Jan 25, 2022

MilesCranmer commented Jan 25, 2022

gkronber commented Jan 25, 2022 •

edited

Loading

MilesCranmer commented Jan 25, 2022 •

edited

Loading

lacava commented Jan 25, 2022

MilesCranmer commented Jan 25, 2022 •

edited

Loading

lacava commented Jan 25, 2022

MilesCranmer commented Jan 25, 2022

MilesCranmer commented Jan 26, 2022 •

edited

Loading

lacava commented Jan 30, 2022

lacava commented Jan 30, 2022 •

edited

Loading

lacava left a comment

lacava Jan 30, 2022

MilesCranmer Jan 31, 2022

lacava Jan 31, 2022

Add PySR & SymbolicRegression.jl to suite #62

Add PySR & SymbolicRegression.jl to suite #62

Conversation

MilesCranmer commented Jan 15, 2022

lacava commented Jan 19, 2022

foolnotion commented Jan 19, 2022

MilesCranmer commented Jan 20, 2022

lacava commented Jan 21, 2022

folivetti commented Jan 21, 2022

MilesCranmer commented Jan 21, 2022

MilesCranmer commented Jan 22, 2022

MilesCranmer commented Jan 23, 2022

MilesCranmer commented Jan 25, 2022

MilesCranmer commented Jan 25, 2022

lacava commented Jan 25, 2022

MilesCranmer commented Jan 25, 2022

gkronber commented Jan 25, 2022 • edited Loading

MilesCranmer commented Jan 25, 2022 • edited Loading

lacava commented Jan 25, 2022

MilesCranmer commented Jan 25, 2022 • edited Loading

lacava commented Jan 25, 2022

MilesCranmer commented Jan 25, 2022

MilesCranmer commented Jan 26, 2022 • edited Loading

lacava commented Jan 30, 2022

lacava commented Jan 30, 2022 • edited Loading

lacava left a comment

Choose a reason for hiding this comment

lacava Jan 30, 2022

Choose a reason for hiding this comment

MilesCranmer Jan 31, 2022

Choose a reason for hiding this comment

lacava Jan 31, 2022

Choose a reason for hiding this comment

gkronber commented Jan 25, 2022 •

edited

Loading

MilesCranmer commented Jan 25, 2022 •

edited

Loading

MilesCranmer commented Jan 25, 2022 •

edited

Loading

MilesCranmer commented Jan 26, 2022 •

edited

Loading

lacava commented Jan 30, 2022 •

edited

Loading