Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor: Remove Cython - Reduce tech debt #346

Merged
merged 105 commits into from
Jul 12, 2024
Merged

Refactor: Remove Cython - Reduce tech debt #346

merged 105 commits into from
Jul 12, 2024

Conversation

eddiebergman
Copy link
Contributor

@eddiebergman eddiebergman commented Dec 18, 2023

This PR attempts to first convert everything to Python, so a type checker can be run across everything as well as use linters properly. This also includes updating everything to pytest.

Note: This PR is unreviewable. The strategy has been to minimally change tests while refactoring to ensure tested behaviour still passes. Some notable exceptions are where functionality was removed or sampling strategy was changed and the expected values differ. Unfortunatly the formatter has rendered them also unreviewable.

Update: All SMAC unittests passed without code changes so it seems mostly backwards compatible.

Some notable changes:

  • No more unbounded Normals
  • No more quantized hyperparameters
  • Sampling is 3x faster, Neighbors 3.5x faster and validation 2x faster (see scripts/benchmark_sampling.py)
  • More code re-use between classes
  • Pure Python, jumping to definition and docstrings should work in editors
  • Defined data-structures to capture heirarchy in conditional spaces, moving the logic out of ConfigurationSpace.
  • Better defined public API for ConfigurationSpace.
  • Better support for custom hyperparameters, i.e. backed by Scipy Distributions (see BetaFloat/Int). Should require a lot less code to do so.
  • Better defined public API for hyperparameters
  • A whole load of deprecations with recommended alternatives
  • Categoricals/Ordinals support arbitrary objects, e.g. no more need to specify Category([True, False, "None"]) (Now handled in follow up feat: Allow arbitrary values as categoricals/constants/ordinals #376 )

TODO:


@eddiebergman eddiebergman changed the title Refactor: Update types for Cython Refactor: Update types for Cython [In Progress] Dec 18, 2023
Copy link

codecov bot commented Dec 19, 2023

Codecov Report

Attention: Patch coverage is 92.06843% with 51 lines in your changes missing coverage. Please review.

Project coverage is 78.91%. Comparing base (93abc5b) to head (2a29120).
Report is 2 commits behind head on main.

Current head 2a29120 differs from pull request most recent head 09beefd

Please upload reports for the commit 09beefd to get more accurate results.

Files Patch % Lines
ConfigSpace/forbidden.py 82.35% 17 Missing and 1 partial ⚠️
ConfigSpace/conditions.py 93.04% 7 Missing and 1 partial ⚠️
ConfigSpace/configuration_space.py 64.28% 3 Missing and 2 partials ⚠️
ConfigSpace/hyperparameters/normal_float.py 88.09% 4 Missing and 1 partial ⚠️
ConfigSpace/hyperparameters/normal_integer.py 85.71% 4 Missing and 1 partial ⚠️
ConfigSpace/hyperparameters/categorical.py 90.90% 3 Missing ⚠️
ConfigSpace/hyperparameters/ordinal.py 91.42% 3 Missing ⚠️
ConfigSpace/hyperparameters/uniform_integer.py 96.29% 2 Missing ⚠️
ConfigSpace/hyperparameters/numerical.py 97.05% 1 Missing ⚠️
ConfigSpace/hyperparameters/uniform_float.py 96.96% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #346      +/-   ##
==========================================
+ Coverage   73.54%   78.91%   +5.37%     
==========================================
  Files          29       45      +16     
  Lines        2831     4767    +1936     
  Branches      629     1005     +376     
==========================================
+ Hits         2082     3762    +1680     
- Misses        632      806     +174     
- Partials      117      199      +82     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@eddiebergman eddiebergman changed the title Refactor: Update types for Cython [In Progress] Refactor: Remove Cython - Reduce tech debt Mar 28, 2024
@sarah-segel
Copy link

FYI: I tested running DeepCave with ConfigSpace 0.8.0 and ran into the following problems when executing :

  1. I run into an error when calling
    from ConfigSpace.c_util import change_hp_value, check_forbidden
  2. It seems like the default values in my configspace.json cause a problem when calling
    from ConfigSpace.read_and_write import json as cs_json
    cs_json.read(self.configspace_fn.read_text())
    where I get:
    __init__() got an unexpected keyword argument 'default'
  3. When calling hp._transform_scalar(value) I get
    AttributeError: 'UniformFloatHyperparameter' object has no attribute '_transform_scalar'

@eddiebergman
Copy link
Contributor Author

eddiebergman commented Jun 27, 2024

@mfeurer, I did a major push on getting the documentation updated to mkdocs. You can view it locally by cloning this branch, pip install -e "[dev]" and running mkdocs serve. Likely another push should get this to the finish line.

@sarah-segel thanks for the report back, I will look into them with the next big push. Do you have any more information on the first one other than "ran into an error"? Second one is likely a bug on my behalf, will investigate. Last one is good to know about. It was a private method so I considered safe enough to remove. I will add it back in with the deprecation warning.

@eddiebergman
Copy link
Contributor Author

@sarah-segel should have been fixed now, can you try again?

Copy link
Contributor

@mfeurer mfeurer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only ten files to go

- name: "Install dependancies"
run: python -m pip install -e ".[dev]"
- name: "Build Docs"
run: mkdocs build --clean --strict
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, is there a reason you removed the pushing of the documentation?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, we moved away from Sphinx and I didn't know how to automate doc deployments given this setup. I will document how to do it in manual way in CONTRIBUTING.md

@@ -49,12 +47,6 @@ jobs:
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
os: ["ubuntu-latest", "macos-latest", "windows-latest"]
exclude:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do this work now?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently so

ORDERABLE: ClassVar[bool] = True # Let ConfigSpace know there is an order to the values

alpha: float
"""Some docstring decsription of this attribute."""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the docstring be further at the top of this class?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, it's alright right here. It puts the documentation right with the declaration.

@@ -49,6 +49,10 @@
from ConfigSpace.hyperparameters.hyperparameter import Hyperparameter


def some_new_function():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do we need this one for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good spot, removed



def center_range(
center: int,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if the difference from the lower to the center is smaller than from the center to higher? Is this an issue?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, roundrobin() will start just giving those from the above_center only after it can no longer yield equally between below_center and above_center

"""Initialize a hyperparameter.

Args:
name:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it necessary to duplicate this information here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not entirely but I don't have it a different way. This is documenting parameters, while the other ones are documenting attributes. I would say it's necessary as it shows up in editors as help information

src/ConfigSpace/hyperparameters/hyperparameter.py Outdated Show resolved Hide resolved
The probability density of the values. Where values are not legal,
the probability density is zero.
"""
# TODO(eddiebergman): Backwards compatible restriction, why this restriction?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be great to know ;)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will leave it in now unless it's a problem at some point

Copy link
Contributor

@mfeurer mfeurer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I finished looking at all files :) Now I need to look at your answer to my previous questions...

src/ConfigSpace/_condition_tree.py Outdated Show resolved Hide resolved
src/ConfigSpace/_condition_tree.py Outdated Show resolved Hide resolved
src/ConfigSpace/_condition_tree.py Outdated Show resolved Hide resolved
src/ConfigSpace/hyperparameters/distributions.py Outdated Show resolved Hide resolved
@eddiebergman
Copy link
Contributor Author

Will be making a release once these tests pass :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment