LinearRegression: add support for multiple targets

[ahendriksen]

LinearRegression did not have support for target vectors with multiple columns previously. This PR adds support.

[tfeher]

closes #3850

[tfeher]

Thanks Allard for the PR! It looks great overall, I just have a few comments.

[ahendriksen]

Thanks for catching the skipped tests. That was an oversight on my part.

[tfeher]

Thanks Allard for the update, the PR looks good to me!

[csadorf]

Forgot to mention in my review, I think the generate_docstring decorator would also require an update, but might be difficult if the change does not apply to all LinearPredictMixin children:

cuml/python/cuml/linear_model/base.pyx

Line 64 in 677a08b

'shape': '(n_samples, 1)'})

[csadorf]

@ahendriksen Thanks a lot for tackling this! 👍 I have provided a few comments. My main concern is with potentially unnecessary conversions, would appreciate if those could be addressed.

[ahendriksen]

On a representative benchmark, this PR speeds up LinearRegression by 20-50x compared to using a loop in Python. This was how multi-target linear regression has been implemented in practice so far.

Results on Volta for the below script show that the new code is faster both when the number of targets is large and when it is small.

loop: 13.24 seconds                       
new: 0.22 seconds                  

# and with fit_intercept=True
loop: 15.02 seconds     
new: 0.20 seconds

# with n_targets = 2
loop: 0.01 seconds
new: 0.01 seconds

# with n_targets = 10
loop: 0.04 seconds
new: 0.01 seconds

import cupy as cp
from cuml.linear_model import LinearRegression
from time import perf_counter as timer
from contextlib import contextmanager

@contextmanager
def time(name):
    # Code to acquire resource, e.g.:
    start = timer()
    yield
    duration = timer() - start
    print(f"{name}: {duration:0.2f} seconds")

n_features = 3
n_samples = 91_000
n_targets = 5_000

X = cp.random.normal(size=(n_samples, n_features))
y = cp.random.normal(size=(n_samples, n_targets))
out1 = cp.zeros(y.shape)
out2 = cp.zeros(y.shape)

# Create linear regression instance that can be reused.
lr = LinearRegression(fit_intercept=False, output_type="cupy", algorithm="svd")

with time("loop"):
    for i in range(n_targets):
        lr.fit(X, y[:, i])
        out1[:, i] = lr.predict(X)

with time("new"):
    lr.fit(X, y)
    out2[:] = lr.predict(X)

[codecov-commenter]

Codecov Report

❗ No coverage uploaded for pull request base (branch-22.12@6500897). Click here to learn what that means.
Patch has no changes to coverable lines.

Additional details and impacted files

@@               Coverage Diff               @@
##             branch-22.12    #4988   +/-   ##
===============================================
  Coverage                ?   79.44%           
===============================================
  Files                   ?      184           
  Lines                   ?    11698           
  Branches                ?        0           
===============================================
  Hits                    ?     9293           
  Misses                  ?     2405           
  Partials                ?        0           

Flag	Coverage Δ
dask	45.93% <0.00%> (?)
non-dask	68.97% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[csadorf]

I think we should also update the tags of the estimator class by adding

@staticmethod
def _more_static_tags():
   return {"multioutput": True}

to the base class.

[ahendriksen]

Thank you for the review! Due to the removal of the conversions, the code became another factor of 2 faster. Apologies for the wip commits. I have a bit of a hobbled workflow when working with pyx files.

[ahendriksen]

Can this PR be merged?

[cjnolet]

This LGTM. Thanks Allard!

[cjnolet]

@gpucibot merge