add utility functions to NX

[tiagodavi]

This PR adds some utility functions to NX.

[josevalim]

Thank you @tiagodavi! @polvalente / @seanmor5, do you think this belongs in Nx? Or some statistics oriented package?

[polvalente]

Thank you @tiagodavi! @polvalente / @seanmor5, do you think this belongs in Nx? Or some statistics oriented package?

@josevalim I think std/var/avg are common and basic enough to belong in Nx. For instance, random variables are often normalized through the standard deviation, and those are Nx vectors.

However, perhaps we could think about moving them to another module.

Speaking of avg (or mean), do we have it yet? Could be a good companion function here.

[josevalim]

We do have mean. @tiagodavi, can you please rename those functions to they have the full name instead of abbreviations? We typically avoid abbreviations in Nx.

[josevalim]

I think the type is aggregation or similar. Pls check what Nx.mean does :)

[tiagodavi]

I'll take a look. thank you

[jonatanklosko]

We should document the exact formula, in particular that we divide by N. In statistical use cases it's often desired to divide by N - 1 when calculating sample variance/stdev, since that gives an unbiased estimate.

For the record numpy.var has ddof option and divides by N - ddof, torch.var has a more explicit unbiased option, while tfp.stats.variance always uses N.

[tiagodavi]

added. good call.

[floatn]

What about zero std value here?

[tiagodavi]

what do you mean by zero ?

[josevalim]

I believe the standard_deviation can return 0 (for example if all values are equal). In this case, this function will error. Are there other formulas to implement standard_scale?

[msluszniak]

Look that std is 0 only and only if all the values are equal. In this case, there is no possibility to achieve unit variance, so the data should be left as-is. And actually, I've searched how it is calculated in Sci-kit learn. They based on this article http://www.cs.yale.edu/publications/techreports/tr222.pdf.

[msluszniak]

But this problematic case is still handled particularly. What is more, they catch the cases when the std is non-zero, but values are almost constant for stability issues. https://github.com/scikit-learn/scikit-learn/blob/7e1e6d09bcc2eaeba98f7e737aac2ac782f0e5f1/sklearn/preprocessing/_data.py#L84

[josevalim]

Yeah, let's give up on the standard_scale/normalize for now then and revisit later.

[polvalente]

What is the definition for the variance of a 0-dimensional tensor? Perhaps we should have a check against that

[tiagodavi]

iex(3)> Nx.variance Nx.tensor(5)
#Nx.Tensor<
  f32
  0.0
>

[josevalim]

Since we may want to add an axes option to those functions, perhaps ddof must be an option too? See how mean is done, we want to accept the same options. :)

[seanmor5]

I am in favor of standard deviation and variance, but I'm not sure standard scale belongs in Nx. It feels more like the responsibility of another library related to data preprocessing

[tiagodavi]

I am trying to do something like this to respect the guidelines, but not sure how to access the tensor over its axes to compute the variance at the end:

elixir
def variance(tensor, opts \\ []) do
    %T{shape: shape, names: names} = tensor = to_tensor(tensor)

    mean_den =
      if axes = opts[:axes] do
        mean_den(shape, Nx.Shape.normalize_axes(shape, axes, names))
      else
        mean_den(shape, nil)
      end

    ddof = Keyword.get(opts, :ddof, 0)
    mean = mean(tensor, Keyword.take(opts, [:axes, keep_axes: false]))

    axes = axes(tensor)

    IO.inspect axes

    tensor[axes] # I am assuming I need to calculate based on its axes because mean and mean_den are based on it.
    |> subtract(mean)
    |> power(2)
    |> sum()
    |> divide(mean_den - ddof)
  end

[josevalim]

Oh, I see. Let’s not worry about the axes version for now then!

[tiagodavi]

@josevalim done. Fixed spaces, removed standard_scale function and added the ddof option to variance, standard_deviation without axes for the time being.

[polvalente]

Suggested change

	ddof = Keyword.get(opts, :ddof, 0)
	ddof = Keyword.fetch!(opts, :ddof)

[tiagodavi]

done

[polvalente]

Suggested change

tensor = to_tensor(tensor)

In this specific case we can just delegate to variance

[tiagodavi]

done

[josevalim]

💚 💙 💜 💛 ❤️