Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't predict from model with more than 256 features #745

Closed
jimfulton opened this issue Sep 4, 2019 · 1 comment
Closed

Can't predict from model with more than 256 features #745

jimfulton opened this issue Sep 4, 2019 · 1 comment
Labels
not a koalas issue

Comments

@jimfulton
Copy link

jimfulton commented Sep 4, 2019 •
edited

When running a model to generate predictions, koalas+mlflow create PySpark UDFs that take an argument per feature column.

PySpark currently has a limitation of 256 arguments per UDF. This limitation seems shallow and easy to hack around. See https://issues.apache.org/jira/browse/SPARK-28978.

Of course, I tried koalas with a model with 500+ features. :)

Unless I'm mistaken (quite possible), this is an easy fix to PySpark. I'm creating the issue here because of the impact on potential koalas users and hoping y'all can encourage the PySpark fix.
@HyukjinKwon
Copy link
Member

This is fixed at SPARK-28978.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
not a koalas issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jimfulton @HyukjinKwon