Domain adaptation Classes

[rflamary]

We should change the domain adaptation Classes to be more sklearn compliant.

Main issues:

• Use CamelCase for classes
• Use init for setting parameters and instead of fit.

@agramfort proposed to Creat new Clases with proper names and begin deprecating the old classes.

I think it is a good move.

[ngayraud]

Good morning!I have coded a Sklearn compliant version of Domain Adaptation that I have been using for months... Would you like to see some of the code?NathalieLe 24 juil. 2017 08:31, Rémi Flamary <notifications@github.com> a écrit :We should change the domain adaptation Classes to be more sklearn compliant. Main issues: Use CamelCase for classesUse init for setting parameters and instead of fit. @agramfort proposed to Creat new Clases with proper names and begin deprecating the old classes. I think it is a good move. —You are receiving this because you are subscribed to this thread.Reply to this email directly, view it on GitHub, or mute the thread.

[rflamary]

Yes @ngayraud we are interested by sklearn compatibility.

Note that we now have more strict rules for contribution now (pep8 for instance) but we would definitely welcome your contribution.

[agramfort]

Hi @ngayraud <https://github.com/ngayraud> I am interested with your sklearn API for DA with @Slasnista we had started something similar for semi-supervised DA. let's see if we can all converge a good API together.

[ngayraud]

Great ! All the work is on the domain-adaptation-scikitlearn-compatible branch. I had created my own class outside github, and I have merged as many lines of code as I could.

[Slasnista]

Hi @ngayraud I am really interested in going into this direction too ! We have started considering this direction as @agramfort said.

We had introduced a BaseTransport class from which all the other DA methods inherited. The main functions where a fit_transform and _fit with parameters X, y, domain like this:

`

def fit_transform(self, X, y=None, domain=None):
    return self._fit(X, y, domain=domain)._transform(X, domain=domain)

def _fit(self, X, y=None, domain=None, method="sinkhorn"):
    """fit : compute transport plan
    """

    # pairwise distance
    Xs = X[domain == 0]
    Xt = X[domain == 1]
    Cost = pairwise_distances(Xs, Xt, metric="sqeuclidean")`

This way, one could do:
ot = OptimalTransportAlgo()
transp_X = ot.fit_transform(X, y, domain)

Maybe we could mix our two approaches ?

[ngayraud]

Absolutely! I have implemented a similar approach for the fit(X,y) and transform(X) methods.  I submitted a merge request, mostly for you to see the entire code. I also agree that the classes should be properly named.However, in my implementation, one fits with the source domain, providing the target domain samples in the class initialization, and transports computes the transport to the target domain when transform (X) is called, so there is no need for a "domain" variable. If the target domain is not provided during the class initialization, the inverse transfer takes place (target to source). Do you have read access to the merge request?Le 24 juil. 2017 17:40, Stanislas Chambon <notifications@github.com> a écrit :Hi @ngayraud I am really interested in going into this direction too ! We have started considering this direction as @agramfort said. We had introduced a BaseTransport class from which all the other DA methods inherited. The main functions where a fit_transform and _fit with parameters X, y, domain like this: ` def fit_transform(self, X, y=None, domain=None): return self._fit(X, y, domain=domain)._transform(X, domain=domain) def _fit(self, X, y=None, domain=None, method="sinkhorn"): """fit : compute transport plan """ # pairwise distance Xs = X[domain == 0] Xt = X[domain == 1] Cost = pairwise_distances(Xs, Xt, metric="sqeuclidean")` This way, one could do: ot = OptimalTransportAlgo() transp_X = ot.fit_transform(X, y, domain) Maybe we could mix our two approaches ? —You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

[rflamary]

Hello and thank you for your proposal.

You both propose nice options, here are some comments.

• It seems to me that a unique fit for estimation the DA is a good thing. Doing the learning in two steps with fit than transform as proposed by @ngayraud is a bit surprising.
• On the other end, X,y and domain seem to be overly complex since you need to concatenate the x samples and provide a domain variable that you need to create also. what about y? does it contain labels from source of target or both? what doe it contain when no target labels are available?

I know that both of you have though about it for a long time but personally I would use X,y, labels_X for fit where X is source domain, y target domain (it works since the transform will go from X to y) and labels are source labels when available. if target labels are available we can also use labels_y.

The thing i propose also have a natural extension to multidomain adaptation since X can be a list of arrays (and their corresponding labels).

What do you think about this?

[ngayraud]

I have been using the fit and transform implementation for a long time, since I am using it in a scikitlearn Pipeline method(are you familiar with it?) . The pipelines require both fit and transform methods to be implemented, and with this implementation I have been able to do cross validations and such. I think it is very handy, especially since I wanted to classify data after the transportation. This is the reason I think setting y as the target domain sample is not such a good idea, since it goes against classic fit methods of scikitlearn. Which is why I give the target domain samples in the initialization... Le 24 juil. 2017 18:47, Rémi Flamary <notifications@github.com> a écrit :Hello and thank you for your proposal. You both propose nice options, here are some comments. It seems to me that a unique fit for estimation the DA is a good thing. Doing the learning in two steps with fit than transform as proposed by @ngayraud is a bit surprising.On the other end, X,y and domain seem to be overly complex since you need to concatenate the x samples and provide a domain variable that you need to create also. what about y? does it contain labels from source of target or both? what doe it contain when no target labels are available? I know that both of you have though about it for a long time but personally I would use X,y, labels_X for fit where X is source domain, y target domain (it works since the transform will go from X to y) and labels are source labels when available. if target labels are available we can also use labels_y. The thing i propose also have a natural extension to multidomain adaptation since X can be a list of arrays (and their corresponding labels). What do you think about this? —You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

[Slasnista]

@ngayraud Oh yes I have seen it ! That's a good point if you don't have the "domain" variable.

• But does it induce a longer runtime when you have large datasets, especially large Xt ?
• My other fear is that if one want to save the TransportObject (with pickle for instance), having the target data as an attribute would induce an increase in the file size.
• For the transform and inverse transform, what about implementing both methods, so that one can chose towards which domain to perform the adaption ?

@rflamary

• I admit that it induces more complexity and that's the real drawback of such a convention
• on the other hand, all the OTDA algorithms would have the same formalism and entry parameters: numpy arrays with the same number of rows (X.shape[0] = y.shape[0] = domain.shape[0]
• this would not depend on their semi or unsupervised aspects.
• we could chain the OTDA methods in sklearn-like pipelines with other preprocessing methods such as StandardScaler, zca whitening...
• for the labels one could use the following conventions (filling the missing labels with -1)
  X = np.concatenate((Xs, Xt), axis=0)
y = np.concatenate((ys, yt, -1 ... -1), axis=0)

So if we perform unsupervised DA and the method does not require any label, we use y=None
Otherwise, we construct y as above, filling the missing values by -1 so that y has the same number of rows as X.

We used afterwards the following fit function header but there is a lot of room for progress and simplicity ;)

def _fit(self, X, y=None, domain=None, method="sinkhorn"):
    """fit : compute transport plan
    """

    # pairwise distance
    Xs = X[domain == 0]
    Xt = X[domain == 1]
    Cost = pairwise_distances(Xs, Xt, metric=self.metric)

    if self.mode == "semisupervised":
        ys = y[domain == 0]
        yt = y[domain != 0]
        yt_train = yt[yt != -1]

        # change cost matrix
        classes = np.unique(ys)
        for c in classes:
            ids = np.where(ys == c)
            idt = np.where(yt_train == c)

            for j in idt[0]:
                Cost[ids[0], j] = 0
    else:
        if y is not None:
            ys = y[domain == 0]

[ngayraud]

Also, multidomain adaptation is a good idea, we could think of how to add it to the implementation for both ways, target to source and source to target!Le 24 juil. 2017 19:16, Nathalie Gayraud <nathalie.gayraud@inria.fr> a écrit :I have been using the fit and transform implementation for a long time, since I am using it in a scikitlearn Pipeline method(are you familiar with it?) . The pipelines require both fit and transform methods to be implemented, and with this implementation I have been able to do cross validations and such. I think it is very handy, especially since I wanted to classify data after the transportation. This is the reason I think setting y as the target domain sample is not such a good idea, since it goes against classic fit methods of scikitlearn. Which is why I give the target domain samples in the initialization... Le 24 juil. 2017 18:47, Rémi Flamary <notifications@github.com> a écrit :Hello and thank you for your proposal. You both propose nice options, here are some comments. It seems to me that a unique fit for estimation the DA is a good thing. Doing the learning in two steps with fit than transform as proposed by @ngayraud is a bit surprising.On the other end, X,y and domain seem to be overly complex since you need to concatenate the x samples and provide a domain variable that you need to create also. what about y? does it contain labels from source of target or both? what doe it contain when no target labels are available? I know that both of you have though about it for a long time but personally I would use X,y, labels_X for fit where X is source domain, y target domain (it works since the transform will go from X to y) and labels are source labels when available. if target labels are available we can also use labels_y. The thing i propose also have a natural extension to multidomain adaptation since X can be a list of arrays (and their corresponding labels). What do you think about this? —You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

[ngayraud]

@Slasnista as a matter of fact, when the transport is source to target, one needs to only store the source and target domain uniform distribution, as well as, the cost matrix. For the inverse transformation, the source domain samples are also needed, it is true. As for the runtime, I don't understand why it would take longer. In practice I have seen no difference.
I agree about the double implementation, I seems cleaner to somehow implement the two directions separately.

[rflamary]

Hello everyone,

@ngayraud I misspoke when I talked about transform, i meant _transport that force you to do a fit with the source and then a transport with the target before doing transform/prediction. Don't worry i'm well aware of how sklearn work and have been using the validation classes for years.
I agree with you that multidomain is interesting, I you want to implement the aggregation method in your paper you are welcome and we will add the reference to the toolbox.

@Slasnista

• Computational complexity is very high in this method and I I believe the recent NIPS paper will stay high for a long time.
• @ngayraud is right that transporting the source sample only require the target samples and the transport matrix but if you want to perform out of sample prediction (Ferradans style) you need to store both. The memory complexity is a problem but storing the data is classical stuff for KNN and people live with it. I would be more concerned with the OT matrix that is extremely long to estimate and a memory hog on large datasets.
• I strongly agree with the use of transform and inverse_transform for out of sample transport
• I agree that OTDA could and should be chainable without other sklearn classes. Thing is I see it aas a pre-processing step since the transform do not change the size of the data. In this sens it should be implemented similarly to StandardScaler except it requires obviously 2 set of training instead of 1.
• Concerning the concatenation of source and targets, I still thing it makes everything more complex since all datasets I know provides you with separate matrices (that you can concatenate with 2 lines OK prett simple). Moreover It does not make sens to perform StandardScaling on the concatenated samples since we have an adaptation problem and the source and target do not follow the same distribution. (possibly with different means). In DA the StandardScaler should be performed independently on each domain which simply forbids simple chain since you have to split and normalize.
• If we keep the X,y,domain formulation though we need to provide simple to use functions that will create those matrices from xs,yx,xt,yt data and vice versa.

The way I see it, OTDA is a pre-processing step that will move your xs sample onto xt. If you want to perform classification simultaneously, I think we should have a different class that provides a predict function and others(classes).