Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Ludwig AutoML Feature Type Selection #1485

Merged
merged 2 commits into from
Nov 17, 2021
Merged

Update Ludwig AutoML Feature Type Selection #1485

merged 2 commits into from
Nov 17, 2021

Conversation

amholler
Copy link
Collaborator

@amholler amholler commented Nov 14, 2021

This Pull Request is intended to improve Ludwig AutoML automatic feature type selection. The change set includes Justin's PR (not yet landed) to improve binary feature type selection as well as additional feature type selection updates to address opportunities observed relative to the types selected for LBT and/or by practitioner(s) on the 12 tabular datasets used to develop AutoML heuristics. The additional feature type selection updates were largely adapted from ideas discussed in the Ludwig AutoML meetings.

The PR is based on the tf-legacy branch, and comprises the following:

  • Justin's PR to improve binary feature type selection
    Classify binary data type only if the values are bool-like. #1473
  • Changes in ludwig/automl/base_config.py and ludwig/utils/strings_utils.py
    ** Refinement of binary type identification to support binary for single distinct values
    ** Refinement of category type identification for small-count distinct values to further specify that those values either not be all numericals or all be sequential integers, the latter suggesting that the numbers were chosen to represent categories.
    ** Refinement of fall-through remaining case type selection as category if sampling finds non-numerical values; else numerical.

The impact of the PR was assessed by comparing its type output for the create_auto_config API with that produced by the current tf-legacy branch code and that chosen for LBT and manually by practitioners for the 12 tabular datasets used to develop AutoML heuristics. Given the positive impact, it is proposed to land this change to Ludwig’s tf-legacy and master branches. The results of this testing are available here: https://docs.google.com/document/d/1nLDbkYtg5J5Xb3sqRF25EGu3O2OOUwOSnbvQKdNhz68/edit?usp=sharing

Copy link
Collaborator

@tgaddair tgaddair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! I like these changes. Similar to @justinxzhao 's PR, we should also think about how to handle this for large datasets, where it may be expensive to carry through all the distinct values in the metadata.

My thought at the moment is we can push down a lot of these computations into the DataSource abstraction so we compute only the derived data from the distinct values (like whether the values are sequential, etc.). Happy to deep dive into this in more detail.

@amholler
Copy link
Collaborator Author

Hi, @tgaddair Thank you very much for your feedback! I believe I have addressed the
scaling issues and the other comments you made on @justinxzhao 's PR (1473). Could
you please take another look? Thanks again!

Copy link
Collaborator

@tgaddair tgaddair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix! This looks good to me!

@tgaddair tgaddair merged commit 91ccb04 into ludwig-ai:tf-legacy Nov 17, 2021
tgaddair pushed a commit that referenced this pull request Nov 18, 2021
Co-authored-by: Anne Holler <anne@vmware.com>
tgaddair added a commit that referenced this pull request Nov 19, 2021
Co-authored-by: amholler <86269492+amholler@users.noreply.github.com>
Co-authored-by: Anne Holler <anne@vmware.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants