Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH - Gets SciKeras script working #394

Open
wants to merge 47 commits into
base: main
Choose a base branch
from

Conversation

lazarust
Copy link
Contributor

WIP still need to fix one test

Reference Issues/PRs

Fixes #388

What does this implement/fix? Explain your changes.

This fixes a recursion error that was happening when dumping a scikeras model.

Any other comments?

@lazarust lazarust marked this pull request as ready for review October 11, 2023 00:08
@lazarust lazarust changed the title Gets SciKeras script working ENH - Gets SciKeras script working Oct 13, 2023
@lazarust
Copy link
Contributor Author

It looks like all the pytest tests are failing with You have exceeded our daily quotas for action: createRepo. We invite you to retry later.

@BenjaminBossan
Copy link
Collaborator

Should be addressed by #398

@BenjaminBossan
Copy link
Collaborator

@lazarust Could you please solve the merge conflict? Regarding the uncovered new line: Would that be solved by adding tests for scikeras?

@lazarust
Copy link
Contributor Author

@BenjaminBossan I've fixed the merge conflict.

Yeah, I believe it would be but I was unsure if we wanted to have tests that included another library like that. Should I add one?

@BenjaminBossan
Copy link
Collaborator

Yeah, I believe it would be but I was unsure if we wanted to have tests that included another library like that. Should I add one?

Yes, it would be good, since that was the initial reason for the change. We have external library tests, see here:

https://github.com/skops-dev/skops/blob/main/skops/io/tests/test_external.py

For scikeras, it won't be possible to add a comprehensive coverage of all possible models, so a simple example should be enough. If it's possible to add a unit test independently of scikeras that explicitly tests weakrefs, that would also be good, not sure how easy it is to do.

Regarding the failing tests, at first glance, it appears to be a change in the model repr in the latest sklearn version, so not related to the changes here.

@lazarust
Copy link
Contributor Author

@BenjaminBossan Sorry this took me so long to get back to. I've added a test to hit that line.

Copy link
Collaborator

@BenjaminBossan BenjaminBossan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks so much for the added scikeras tests. I just have a small request to improve them a little bit.


pipeline = Pipeline([("classifier", clf)])

dump(clf, "keras-test.skops")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of just dumping, could we please do a cycle of dumps and loads, similar to the other tests?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! I now have it dumping the model, loading it back in and comparing the results.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get rid of dump completely in favor of dumps? That way, we also don't need to care about cleaning up any files created during the test.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed! Sorry about that. I didn't realize the difference between dumps and dump 🤯

@BenjaminBossan
Copy link
Collaborator

@adrinjalali The tests for sklearn nightly are failing because the model repr was changed (not sure why). Normally, we could fix that by having the test check the sklearn verison, but this is a doctest. Any idea how it can be fixed, short of skipping it whole?


X, y = make_classification(1000, 20, n_informative=10, random_state=0)
clf.fit(X, y)
dumped = dumps(clf, "keras-test.skops")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
dumped = dumps(clf, "keras-test.skops")
dumped = dumps(clf)

2nd argument to dumps is the compression level. Honestly, I'm surprised that this didn't raise an error.

@BenjaminBossan
Copy link
Collaborator

Ugh, the list of trusted modules is giant now :D I guess it's related to the tensorflow change. Could you please explain why that was necessary? Also, we now get this error on CI:

E AttributeError: module 'scikeras' has no attribute 'wrappers'

@lazarust
Copy link
Contributor Author

@BenjaminBossan Yeah, sorry I realized I had marked the test method as a @pytest.fixture 🤦🏽, so the test wasn't ever running (which is why it wasn't erroring when switching from dump to dumps). I'm hoping to get the test fixed and cleanup that list of trusted modules today.

@lazarust
Copy link
Contributor Author

@BenjaminBossan After staring at this all day, I could use some help lol. For some reason, there's some infinite recursion happening when constructing the tree that I can't figure out why.

Initially, I thought it was due to the CachedNodes in the tree, but the construct method isn't getting hit for those. I've gone through the tree and didn't see anything outrageous so if you could take a look that'd be great!

@adrinjalali
Copy link
Member

I'll have to check when I'm back. Still off till end of November. But I remember CacheNode caused some issues when I was working on it, partly due to small integers and other common objects having the same id in python.

information for CachedNodes

I'm wodering after looking at the types of a lot of the CachedNodes if there's something weird happening with `None`
@lazarust
Copy link
Contributor Author

@adrinjalali I got it working for older versions of Python other than Windows on 3.8 (I think the Ubuntu 3.9 failure was just a codecov issue). Could you help me figure out how to get it working? Downgrading tensorflow to 2.11 just breaks more versions.

@adrinjalali
Copy link
Member

One of the main reasons we support 3.8 was colab, which now supports 3.10. I think we can drop 3.8 alltogether.

Would you be kind enough to submit a PR to remove 3.8 support?

@lazarust
Copy link
Contributor Author

@adrinjalali I created this PR to remove 3.8 support #418, but I'm running into an issue where pip install scikit-learn-intelex isn't finding any valid versions only on MacOS (this is happening both in the GH action and locally for me). I'm still looking into a fix

@BenjaminBossan
Copy link
Collaborator

MacOS issue could be this: actions/setup-python#850

@adrinjalali
Copy link
Member

Intelex is very brittle, and I don't think it's used much if at all. I'd be in favor of removing it. It was only to handle the inference side and HF inference is not that well maintained anyway.

@BenjaminBossan any objections to remove it?

@BenjaminBossan
Copy link
Collaborator

Works for me.

@adrinjalali
Copy link
Member

@lazarust you've been very kind here. Would you remove sklearn-intelex in a separate PR too?

@lazarust
Copy link
Contributor Author

lazarust commented May 5, 2024

@adrinjalali Done! I removed sklearn-intelex in #420

@adrinjalali
Copy link
Member

@lazarust let me know when the CI is green and ready for review here again.

@lazarust
Copy link
Contributor Author

@adrinjalali Sounds good, it may take me a couple of days to get to it. It looks like there's something wrong in tensorflow with how it's using google.protobuf

@adrinjalali
Copy link
Member

The errors seem like a bunch of deprecation warnings due to some updates in dependency packages maybe?

@lazarust
Copy link
Contributor Author

@adrinjalali I created an issue in tensorflow to discuss this and make sure I wasn't missing anything tensorflow/tensorflow#68194.

It seems like the best way forward is to ignore the warnings for now until Tensorflow supports protobuf 5.0+. Does that sound good to you? Ignoring warnings doesn't sound like the best thing I'm just unsure of a better way forward.

@adrinjalali
Copy link
Member

Ignoring warnings in cases where we know why they're happening and with a comment as when the ignore statement should be removed is a normal practice indeed. Thanks for the follow up work on the TF side.

@lazarust
Copy link
Contributor Author

@adrinjalali This should be ready for you to look at again. The failing tests seem to just be a blip with codecov

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bug: dump fails on keras models with RecursionError: maximum recursion depth exceeded, after calling fit
3 participants