[WIP] Return basic statistics to Base

[ararslan]

This reverts "move cor, cov, std, stdm, var, varm and linreg to StatsBase (#27152)," commit 746d08f. Fixes #27374.

[KristofferC]

Stuff like linreg should not be in Base.

[ararslan]

Yeah I'm fine with removing linreg, but definitely not with removing the rest of the stuff. Though linreg was in LinearAlgebra, not Base.

Edit: Added a commit that removes linreg.

[StefanKarpinski]

cor, cov, stdm, var, varm don't belong in Base either. Literally the only one of those functions that is truly objectionable to move out of Base is std and honestly, that's a very unfortunately short name and separating std from stdm seems pretty weird. So we could keep std and stdm in Base... but really? Why don't we have mean absolute deviation functions in Base or any other dispersion statistics?

[sbromberger]

@StefanKarpinski

Why don't we have mean absolute deviation functions in Base or any other dispersion statistics?

Because these are generally unuseful to laypeople. Standard deviation, on the other hand, is common enough to be considered "general purpose".

@KristofferC

Stuff like linreg should not be in Base.

OK, I might agree with you in this particular case, but what are the criteria that we (as a project) have set for what gets put in Base, what gets moved to Stdlib, and what is the responsibility of a third-party package? It seems arbitrary to me that std is now third-party while mean and median are in Base. Was there any reasoning that led to this decision, or was it just an unfortunate result of a drive towards slimming Base?

[Nosferican]

I would favor having statistics.jl moved to StatsBase, but moving half of it and keeping half in Base seems very arbitrary.

[ViralBShah]

I tend to agree with @StefanKarpinski here that retaining std makes sense (maybe stdm too) - and it meets @sbromberger's definition of common usage.

I don't understand though for necessarily including something in stdlib. Realistically speaking, to do things, you do need dozens of packages anyways. We are already working on moving SuiteSparse and ARPACK and related capabilities out of stdlib. The more of this stuff is in packages, the easier it is for maintainers to evolve those capabilities. If anything, I think having stats capabilities spread in three places - Base, stdlib/Statistics, and StatsBase could end up being frustrating.

I think having common things in Base (that everyone would want, and I do think std falls in that category) and everything in StatsBase is the right thing.

[juliohm]

Sharing my point of view about this "slimming effect" that we've been experiencing as users and package developers.

I think the real issue here is the lack of a formal specification of what is "in" and what is "out", and most importantly a proper organization of what is left out. Right now many things that were slimmed out of Base got a somewhat random home in a relatively huge package. Take std for example. A basic operation that many would like to have access to without having to depend on StatsBase.jl. Maybe StatsBase.jl is not really a "Base" package anymore? Can it be shrinked further?

I would never complain as a user/package developer to import a separate package to have std if that separate package was indeed minimal. The definition of minimal package is also subjective, but you get the point.

The devs did a good job purging out things that are not essential to the language, and we are now struggling to get them in a place that makes sense and is minimal. Coming up with this set of minimal domain-specific packages is the challenge.

[sbromberger]

to do things, you do need dozens of packages anyways.

This is not my experience at all, and it makes me sad that it seems to be an accepted way of using Julia. Again, I am vehemently opposed to the “npm-ification” of Julia. Perhaps I’m an outlier, or maybe I’m reading too much into this statement, but I have a fundamental aversion to it.

[juliohm]

We could start with some simple rules of thumb like "Base" packages can't have dependencies in their REQUIRE file. This is the current list of dependencies in StatsBase.jl:

julia 0.6
DataStructures 0.5.0
SortingAlgorithms
Compat 0.61.0
Missings

As package developers, we have little control of dependency growth unless we establish some conventions like the above. I propose to call a package "Base" if it only depends on Compat.jl and maybe other packages like Missings.jl (which BTW I think should be part of Julia Base as a too basic type, but that is another issue). Everything else is not considered "Base" and I would be skeptical to add it as a dependency in my projects.

[dpsanders]

I have no idea what stdm or varm are, which seems like a good reason not to have them in Base.
Or should they actually be methods of std and var?

[nalimilan]

As package developers, we have little control of dependency growth unless we establish some conventions like the above. I propose to call a package "Base" if it only depends on Compat.jl and maybe other packages like Missings.jl (which BTW I think should be part of Julia Base as a too basic type, but that is another issue). Everything else is not considered "Base" and I would be skeptical to add it as a dependency in my projects.

Missings.jl is just a Compat-like package for 0.6 and not needed on 0.7. DataStructures is only used in one place and that dependency could probably go away. Finally, I think SortingAlgorithms is worth adding to the stdlib.

Overall StatsBase isn't that big and it's pure-Julia. See #27152 (comment) for a list of features it includes. I'm not opposed to having std in Base since that's a pretty basic function, but for all slightly more advanced functions I don't think it's a problem to depend on StatsBase (whether it lives in the stdlib or not).

[JeffBezanson]

If you want objective criteria for what should be in Base, the closest I can come is "as little as possible". It should contain only functions it would be hard to imagine writing any julia code without. Anything else is just arguing about what is "commonly used" enough. It's fine if people want to have those debates; just be aware that's what you're signing up for.

[oscardssmith]

I think a lot of the contention here is coming from the fact that statsbase isn't in stlib. While I agree with @JeffBezanson that Base should contain as little as possible, I think that one of the things python shows us is that the standard libraries should be the home to as much as possible as long as it is useful, robust, and reasonably performant.

[Nosferican]

From a developer perspective, I favor moving away from stdlib to just packages (easier to maintain, easier for CI, etc.). From a user perspective, I also prefer to have them as packages since it makes explicit my environment and for anything beyond trivial stuff, it would be installed and loaded as dependency when using end user targeted packages.

[ViralBShah]

@sbromberger I think calling this npm-ification of Julia is a bit unfair. Julia packages tend to include larger chunks of functionality. On the dozens of packages, I find myself at least wanting Plots and Gadfly on any machine I am running Julia on, and you are quickly into quite a few packages. I don't think my experience is uncommon.

@dpsanders I was wondering what stdm was too and had to look it up, and then remembered it. Handling them as methods of std seems reasonable to me, but the stats folks may want to chime in.

Whether something is in stdlib or a package is not the same discussion as What should ship in the default download of Julia, even though today the two are connected. Perhaps this is a conversation for JuliaCon, and we might want to have a BOF session on this topic.

[sbromberger]

So, since this discussion came out of a gripe I made in slack, perhaps I can help refocus the discussion. Right now, the following seems to have occurred, resulting in an undesirable situation (at least from my perspective) - perhaps I've misunderstood part of it.

1. a "common" function (std) was moved from Base to a third-party package, StatsBase, sometime between 0.6.2 and 0.7.
2. the plan is to move StatsBase to the stdlib sometime soon, but until then,
3. anyone who wants to use std in their library now must incur the cost of another third-party dependency in REQUIRE, and
4. must then change (with a retag) the code to drop that dependency once StatsBase is in stdlib, because
5. there is apparently a desire to avoid moving things from Base to stdlib directly at this time.

It's the combination of 3 - 5 that is causing me grief. We've spent a TON of time and effort trying to reduce the number of dependencies that LightGraphs and the LG ecosystem has. This was originally "reduce the number of packages that get installed by Pkg.add("LightGraphs") on a clean system", which was originally 44, but got down to 7 or 8 after a month or so of work.

The move of functions from Base to stdlib makes this a bit more complex, since we now can't use "number of import/using statements" as a metric, but rather, "number of packages in REQUIRE, plus their dependents", since the move to stdlib increases the former but doesn't really represent new dependencies.

So what we've done is reimagined our "ecosystem fragility" as a function of "what are we depending on that's not core Julia?" This is important to us (me) because I am seeing a trend over the past 3 years of abandoned / unmaintained packages, and we've been bitten by this directly. In some cases it's just because the package is mature and there's not much left to do; in others, the maintainer has moved on to other things. But it is a higher risk that a third-party package will stagnate than a stdlib package will, so our preference is for the latter over the former given that we've got to change our thinking about Base.

Thus, we're now faced with a decision:

1. incur the cost (both in terms of development [admittedly small], complexity and fragility) of including a third-party package in order to use a function that used to be in Base, and then incur the cost of moving it back to stdlib at some point in the (hopefully near) future;

2. rewrite std and incorporate it in LightGraphs (we've done that for some other things where the dependency cost was too high to justify a REQUIRES just to get a single function); or

3. rewrite our code to eliminate std altogether. This is actually what we've wound up doing over the past couple of days, so we don't have to make a suboptimal decision.

So, from our perspective, the initial problem has been resolved by doing something else entirely; but the issue still remains that in the future our strong preference is to prefer Base / stdlib over third party wherever possible. Our developer guidelines (fourth bullet) need to be updated but the essence still holds, as in order to keep LightGraphs as a "core" library for graph analysis, we need to make sure it's not difficult for end users to use, but just as importantly, for developers to incorporate, adapt, and extend.

Julia Core team, please keep this in mind as you make further decisions about what lives in Base, what lives in stdlib, and what gets moved out to third-party, and the process by which any migration happens.

[chethega]

From a user perspective, my test for Base exports would be: Am I supposed to read the manual for this function before starting to code julia? If not, then it wants to be in a separate namespace.

But I agree that juliaLang should be responsible for maintaining/curating basic necessities like standard deviation; that is, NPM left-pad is a catastrophe to learn from, and a fracturing of dependencies is the worst thing that could happen. But if there is a large majority of users that can get away without knowing of a function's existence, then it only pollutes their namespace and disorganizes the manual.

I would be entirely happy if these things moved to a submodule, like Base.Order.

[ViralBShah]

I think everyone agrees that std will go back to Base, as a common function like mean. Does that not resolve the immediate issue?

Apart from that, yes, I do acknowledge the larger philosophy of burden on package writers, but perhaps we can focus the conversation here on what is best to solve the current issue - being the specific functions in this PR.

I really also don't like how folks are bringing up NPM here - as I think it does a disservice to other open source developers (after all it is very easy to sit in ones armchair and criticize things), and it tends to over-simplify the issues we are trying to sort out.

[sbromberger]

I think everyone agrees that std will go back to Base, as a common function like mean. Does that not resolve the immediate issue?

The immediate issue from LightGraphs' perspective has already been resolved; that is, even if you choose to do nothing, we have worked around the change.

I really also don't like how folks are bringing up NPM here - as I think it does a disservice to other open source developers (after all it is very easy to sit in ones armchair and criticize things), and it tends to over-simplify the issues we are trying to sort out

I think this criticism is unfair. NPM is a classic case of a new approach to dependency management that has seen some very strong drawbacks in implementation (left-pad being one of them; reliance on a central repository being another; fragmentation of that repository being a third). There are lots of advantages to an NPM-like system for Node developers, but I don't think it carries to Julia developers. This is not a criticism of NPM without context, which is what I gather you mean by "armchair criticism"; it is a plea to the Julia team not to create a repository structure that has all the well-known shortcomings of NPM without some significant offsetting benefits. I'm happy to discuss elsewhere if it's a distraction to the current issue.

[JeffBezanson]

I believe with Pkg3 stdlib packages do need to be listed in Project.toml, so your steps 3-5 of first adding and then removing a dependency should not occur.

[ViralBShah]

I would like to point out that we are not moving each function into its own package or splitting things without reason. Historically, we shoved a lot of stuff in Base, before we had a well functioning package manager.

I do agree with the general principles pointed out. I also personally had a similar reaction when we first started this process, but quickly realized it is actually better to group functionality in packages where developers vested in that functionality can own it and can move faster than base.

[ViralBShah]

How about we keep std in StatsBase for a while for now and see how it feels? We can certainly move it to Base before 0.7 if that is what we want, but perhaps develop a feel for it in the meanwhile.

[sbromberger]

How about we keep std in StatsBase for a while for now and see how it feels?

Because it's inconsistent with mean and median, which are in Base. I'd favor moving all of them into a stdlib package, but moving std into a 3rd party package seems like a really bad idea.

[ViralBShah]

I was personally on the fence about moving std out. Let's wait a little bit to gather other feedback here and then modify this PR accordingly. I would be ok for all three (mean, median, std) to be in Base or Stdlib (but together), since they are quite basic and almost everyone knows them.

[sbromberger]

I would be ok for all three (mean, median, std) to be in Base or Stdlib (but together), since they are quite basic and almost everyone knows them.

I was under the impression that it was impossible at this point to move things from Base to Stdlib directly without going through a third-party package. Is this wrong?

[ViralBShah]

I would be onboard with std. My personal list is mean and std in Base, and everything else StatsBase.

[juliohm]

I would like to clarify (as a user and follower of the developments) that the initial goal of being a language for scientific computing has changed along the way. Nowadays, it is incorrect to give Julia this label.

We can either accept the new wave of generality and leave all the statistical functions outside of Base for consistency, or regain the title and export a set of names that anyone doing scientific work would expect to have coming from numpy, octave, matlab, etc.

If all these statistical functions are collected coherently in a package Statistics.jl, it shouldn't be a big problem for users to type this single line before they start their work. The issue is more about dependency increase for package developers, which will have a harder time with keeping their packages small and portable.

[StefanKarpinski]

@juliohm, you are not the arbiter of what does or does not make something a scientific computing language and that kind of rhetoric does not help make your point.

[juliohm]

Sorry for the noise, I am just sharing my feelings about these changes. Won't do it again.

[KristofferC]

mean + std in Base seems reasonable.

[Nosferican]

I discussed the thread with a co-worker and here is a summary of some points from the conversation:

• Julia language as a scientific/computing language/ecosystem might benefit from having both StatsBase and a tabular data representation (e.g., DataFrames) as standard libraries (e.g., JuliaComputing)
• From a security perspective, if Julia is being used in industry and whatnot, having those capabilities shipped with the language might be lot better from a security/permission standpoint (maybe this will be addressed with Pkg curated registries). For example, when using Docker images.
• I believe the "best" decision should be informed by these peripheral concerns (will StatsBase morph to a stdlib/Statistics, will curated registries solve the security/permission issues for packages not shipped with the language (e.g., preferred versions rather than vanilla like Microsoft R Open to R or conda Python distribution)?
• My personal preference is to keep the code topic organized rather than split them by how layman these might be. This is also extremely helpful from a developer maintainer perspective. For example, easier to find trig functions in one rather than by widely-used and more advanced functions.

[ViralBShah]

Those are all valid points. The only argument being that mean and std are functions that most software engineers know about (even non-numerical programmers), much more so than all the esoteric (from a non-numerical programmer perspective) trig functions, for example.

[JeffBezanson]

coming from numpy

...where you have to type import numpy to do anything like this.

As long as we're going to have mean, median, and quantile in Base, I won't fight including std as well. But I just don't get why being in Base is such a big deal. If your requirements disallow using packages, for whatever reason, it seems to me std is the least of your problems.

[StefanKarpinski]

Moving std back to Base seems reasonable. Can we lose stdm then and replace it with a keyword arg to std supplying a known mean value? Separating std from stdm seems weird but stdm does not seem like it belongs as its own name in Base.

[StefanKarpinski]

@juliohm:

Sorry for the noise, I am just sharing my feelings about these changes. Won't do it again.

Your input and opinion are welcomed. If you have a solid argument for or against something, please make it. Please do not make technical choices that you disagree with into "historical disasters" or epic existential threats to the language (because it is no longer a true scientific language). That kind of rhetoric actively hinders having a reasonable, civilized debate about what clearly is a subject where different people, many of whom have been actively involved in scientific computing for a long time, can have divergent opinions. By painting the situation in terms of the people who want Julia to be a scientific language (those who agree with you), versus those who don't (those who disagree), you are implicitly saying that all of the people who disagree with you are not real computational scientists and don't care about scientific computing. In the inner products thread there were many posts where you similarly stated or implied that anyone who disagreed with you was either not a real mathematician or didn't care about correctness (amusingly enough when arguing with Steven Johnson). That's simply not a good stance to approach a debate from. Assume that we are all reasonable people who want Julia to be the best language for scientific computing that it can be (and great for other things too), and that if you make a good case for why your way is better, we will listen to it.

[juliohm]

Thank you for the wise words @StefanKarpinski , I will try to organize my thoughts better in the future. Sometimes, however, I feel that there is reluctance from core devs to think neutrally about new ideas proposed by users and package developers. I am not the first to complain about this as you know, and there are many cases online of users blogging about these problems, which are real. That thread with inner products is a good example where a very decent idea was almost killed from the beginning without clear justification, and I would argue that it unrolled in that long discussion just because of the feedback and comments that I got with no scientific basis, and just personal taste. It felt to me like the the idea was being evaluated with a mindset like "He is not from the pack, why do we have to listen to him?" By all means, we are all here with a shared goal to have a good language, and to design it to the best extent possible. More impartial evaluation of new issues contributed by external users would make things a lot better.

[Nosferican]

My grain of salt in the above is that usually the misunderstanding are from a lack of sufficient nuance. It is not the first (nor will it be the last) time that someone has turned a decision point into a dooming the language scenario or a core developer has shut it down without sufficient nuance as to why the language / framing is disruptive and counterproductive (which can seem unjust / harsh and shut down a potential contributor). However, in most cases the nuance has taken place afterwards (such as in this case) and clarifies the matter improving the community environment. That is something inherent in the system as human beings and hopefully it continuous to improve. Sometimes, I will have issues closed before having time to address follow ups that lead to actual issues addressed, but I totally get why some responses might be a bit rushed (e.g., closing this PR/issue for instance when the discussion has evolve in spite of being closed) due to the need to move through issues as developers.

[JeffBezanson]

I can't help but notice some irony here: some people object to adding a dependency on StatsBase, but their solution to that is to force every julia program to depend on some statistics functions via Base.

[ViralBShah]

The reason this PR is closed is because the discussion for moving everything back from StatsBase is not open. The only thing we are considering is std. Perhaps enough has been discussed... We need a PR.

[Nosferican]

Final vote from my part is that mean and std should be in the same location. I believe it would be best for quantile, median, var, skewness, kurtosis, cov, cor, etc. (get rid of linreg). to be in the same location wherever these end up (Base, stdlib, a Julia ecosystem). Would be nice to have StatsBase statistical weights in the same place too.

[StefanKarpinski]

I think @Nosferican's split is a good one. The only thing we need to get there is to merge stdm into std and move it back into Base. Everything else can stay in StatsBase.

[ViralBShah]

Yes, I'd be up with having the stdm functionality folded into std, perhaps with a keyword argument, but not having it as a separate function in Base.

[ararslan]

Kind of weird that std and var would be separated, IMO, but otherwise that seems like a logical split.

[ViralBShah]

We still have median and quantile in Base. If people want std and var together, which is also reasonable, then perhaps they all need to go to StatsBase.

[StefanKarpinski]

Part of the problem is that std and var are very short, generic names to be claimed in Base. The names median and quantile couldn't really mean anything else, so it's far less problematic. There are quite a few things that var could mean as well as std.

[nalimilan]

Implementation-wise, it would be silly to have std in Base but not var given that the former needs the latter. But I agree median and quantile would better live in StatsBase.

[ararslan]

std -> stddev, var -> variance? ¯\_(ツ)_/¯

[Jutho]

As somebody who almost never uses statistics (and had to check only to find out that StatsBase was installed as a dependency), I would hope that std(dev) remains easily available for quick exploration. My main problem with StatsBase, crazy as it may seem, is a dislike for abbreviated names for packages. I don't mind using LinearAlgebra and using Statistics. But using StatsBase looks awfully unintuitive and is harder to type then using Statistics because of the capital B following the letter s, which makes my left hand hurt.

[gabrielgellner]

I don't personally love the move to stddev and variance, as it feels like it breaks a naming convention (from numpy / matlab, without going to another one like R's sd / var, or Mathematica's name it in full) to add a new arbitrary one. But a larger question: would this mean we would in the future have both spellings of the name, depending on what packages are loaded, or that this would be the new names (in Base I guess...) for these in Julia?

Finally as to the points made about lean vs fat standard libraries is the point made about copying functions into a package to avoid a dependency something that happens in these more modular environments (i.e. npm etc) more often? That seems so scary to me from a reuse perspective, and code idiom perspective. Seems like some tough trade-offs between given the language a built in feel with enough verbs from the standard library that it has some reach to basic packages, versus having too fat of a base distribution which might hamper some domain applications (that require small install footprints). Also having all the documentation distributed across packages is a real pain point as it makes everything feel less integrated.

[StefanKarpinski]

There’s a choice here:

1. Use very short, overly generic, highly overloaded names var and std and have those names in a stats-specific namespace.

2. Have these functions in Base with longer, more descriptive, less overloaded names like variance and stddev.

Refusing to do either and insisting that the only acceptable solution is having super short names and having them in Base strikes me frankly as pretty unreasonable, yet that is the position that seems to be taken by many in this discussion.

[Nosferican]

I prefer one.

[gabrielgellner]

I am not sure it is that clean. Of course, in a perfect world you could pick one or the other, but for really every math function name (i.e. exp, sqrt, sin, cos, I [identity object] etc) we choose a short overly generic name that gets auto imported. At this point these are simply idioms and for them to mean something else hurts the brain. I see the trade off, but we are being just as inconsistent not having to do using Math to get many of these super common features, like one does in python, C etc. I don't think anybody perfectly does 1 or 2.

What we seem to be doing is singling out var and std from the above list of many such common math functions and potentially giving a non idiomatic name because we don't want them in the base namespace. I can see why this might be done, but it is hardly such an unreasonable or uncommon position as you make it sound.

[JeffBezanson]

That in large part comes from exp, sin, etc. being traditional in many languages, much more so than std and var. sin and cos are found even in really minimal environments like micropython and small scheme systems.

I also think the namespace issue is not the main issue. Being in Base does not prevent other packages or code from using those names for something else, though it's often discussed as if that were the case. On the other side, keeping the functions in Base but renaming them would not fix anything as far as I'm concerned. What matters more to me is where the code is located, for purposes of developing and versioning it, and for purposes of removing it as a dependency for people who might want to build minimal executables at some point in the future.

[gabrielgellner]

I agree that std and var is less common (likely just because it is not in math.h :) ), but I think the naming issues are the same, and std / sd and var, but when they exist in a base language they are almost always one of these versions (though I do see that python has added the variance, stddev version to their standard library ... which is ugly given the numpy names ... but so it goes). Mostly this was in response the @StefanKarpinski point about the two choices which focused on the namespacing, variable name length trade-off.

For the code location I do think it has a end user impact as well though which is likely why this always brings up so much discussion when any new piece is removed from base. The rise in using statements in my own code does change the feel of the language, and downloads the knowledge of what are the basic building blocks of the language onto the end user which can feel bad. I feel this ship has sailed in Julia, but the distributed standard library does make Julia feel less integrated, even if that is only from a new or less savy users perspective.

[lstagner]

I'd rather have longer names for std and var than to not have them in base at all.