[EPIC] Launch IbisML

[lostmygithubaccount]

Acceptance criteria: IbisML is improved and publicly launched with a blog.

TODOs:

• investigate integration with https://github.com/ydataai/ydata-profiling

[gforsyth]

Can we define "release" for this? Asking because there are releases on PyPI

[lostmygithubaccount]

true, let me just edit -- we've "released" but we haven't really launched in any public way, that's what this is to capture

[lostmygithubaccount]

Weekly Update:
2024-01-18

• no update

(copying over last week's update on behalf of Gil)

[gforsyth]

No update

[gforsyth]

• Meeting (2024/02/01) to discuss direction and planning for IbisML

[jcrist]

• Met to plan out user stories, sync on current status and design.

[jcrist]

Weekly Update:
2024-02-15

• Continued knowledge transfer to (former) claypot team.
• Continued research and planning out scope and design for a new redesigned ibisml package.

[lostmygithubaccount]

Weekly Update:
2024-02-22

• see feat: support more parts of end-to-end ML workflow #19 for continued discussion
• Jim working on a refactor

[jcrist]

Weekly Update:
2024-03-07

• Refactor PR is up but not yet merged.

[jitingxu1]

Weekly Update:
2024-03-21

• data preprocessing transformation priority: feat: preprocessing transformation priorities #32
• Create issues for P0 transformation
• Create issues for other tasks: test coverage, code clean up

[jitingxu1]

Weekly Update
2024-03-29
PR in review
#48
PR in progress
#34

[jitingxu1]

Weekly Update
2024-04-03

Demo is in progress

PR in review

• feat(steps): implement zero-variance filter #48
• feat(steps): add transform for polynomial features #53

PR in progress

• feat: Target Encoder implementation #34

[jitingxu1]

Weekly Update
2024-04-11
Demo

• Internal demo on 2024-4-11

ML Preprocessing steps implementation

• Merged
  • Zero-variance filter
• Review
  • Polynomial features(resolved feedbacks and wait for new review): feat(steps): add transform for polynomial features #53
  • KBinsDiscretizer: feat(steps): implement DiscretizeKBins transform #57
• In Process
  • feat: Target Encoder implementation #34
  • feat: Outlier - Impute and capping #45

[jitingxu1]

Weekly Update
2024-04-18
Demo

• Did internal demo using duckdb on nycflights data, collected feedbacks for the incoming demo
• Booked the company tech demo on May 2nd.

Backends:

• Ran ML modeling use ibisml and sklearn on Theseus, it works but some steps have issue, will keep working on this

ML Preprocessing steps implementation (P0)

• Merged
  • Zero-variance filter
  • Polynomial features
• Review
  • KBinsDiscretizer: feat(steps): implement DiscretizeKBins transform #57
  • Outlier: feat(steps): univariate outlier detection #68
• In Process
  • feat: Target Encoder implementation #34
• todo
  • Ratio Variable generation [Will pause this one and Prioritize the demo work ]

[jitingxu1]

Weekly Update
2024-04-24
Demo

• Run the same ml workflow on Theseus, detected some issues, and did some fix and raise the issues to the stakeholders.

Newly Merged PRs

• (feat)Outlier: feat(steps): univariate outlier detection #68
• (bug)Fix one hot encoding nan and cast Null to int issue

Review

• KBinsDiscretizer: feat(steps): implement DiscretizeKBins transform #57
• feat: Target Encoder implementation #34

[jitingxu1]

Weekly Update
2024-05-02

Demo:

• Company Demo: Executing a machine learning workflow using ibisml and scikit-learn with various models (scikit-learn, XGBoost, PyTorch) across different badkends including Theseus, BigQuery, and DuckDB

Tech

• update ibis dependency to 9.0
• merged two new preprocessing steps: KbinsDiscretizer and target encoding
• add model picker to examples
• support torch
• restructure ibisml website and add newly steps to documentation
• fixed bugs:

[jitingxu1]

Weekly Update
2024-05-15

We majorly focus on release work for the past two weeks, including planning, bug fix and codebase refactor.

Done:

• code refactor
  • rename it to ibis_ml
  • make moudules under step private
• make websites and docs prettier
• bug-fix

In Progress

• Add support matrix for different backends
• Unit tests for different backends. [we may only use the support matrix to tell users how many steps are available for the specific backend, this effort probably will not continue]

[jitingxu1]

Weekly Update
2024-05-23

Done

• Submitted 30-min talk abstract to ODSC-West
• Initial version of ibisML support matrix - link, plan to better show the information to the users, and prepare the 2024 table contest

In Progress

• feat(core): implement Recipe and Step printing #101
• feat(core): visualize recipes in sklearn pipelines #100

[jitingxu1]

Weekly Update
2024-05-30

Done

• ibisML version 0.1.0 pypi release
• Better website
  • publish support matrix
  • Make the better tutorial: https://ibis-project.github.io/ibis-ml/tutorial/pytorch.html
  • How to build your own transformer docs(website): add custom transformation to website #116

Todo:

• blogpost
• Add the remains to FAQ in docs(website): create an FAQ #92 docs: make website launch-ready #69

[jitingxu1]

Weekly Update
2024-06-10

Done:

• Finish a kaggle project with IbisML
• External clients exploration [Ongoing]

Todo:

• Prepare the presentation for risingwave
• Polish the kaggle project for a blog and examples
• Improve website
• fix bugs

[deepyaman]

Weekly Update
2024-06-20

Done:

• Delivered lightning talk at PyData London
• Fixed issues to enable running the tutorial on Postgres (in the form of upstream Ibis enhancements)
• Website content enhancements

In progress:

• Resolve button formatting issue on website

To do:

• Add FAQ section to website
• Improve rendering of support matrix