Begin the dissolution of libextra #11787
Conversation
|
|
|
That is true, I think that rustdoc is going to need a broader solution for integrating the documentation of multiple crates together. For now, we can try to update The good news is that this will continue to generate, test, and upload documentation, you'll just have to know where to go to find it (for now) |
|
Just to be clear, if after this PR lands I want to use extern mod flate;
use flate::something;Correct? |
|
That's the idea! |
|
For future visitors to this page, could you explain why people wanted to dissolve |
|
The current pattern of the rust compiler is to have very large libraries with lots and lots of modules inside them. This has become less than desirable for a few reasons
Essentially, whenever you link to a crate, you should be planning on using the majority of the crate. Currently, this is not at all the case of libextra where each module is very different in functionality than the next. All of the contents of libextra are quite valuable, but they're arguably more valuable on their own. Functionality like the We would like to start fostering a pattern of smaller, more focused crates than we've been doing in the past, starting with libextra being broken up. These smaller crates will allow for greater individual expansion in the future instead of maintaining one monolithic crate. Essentially, we want to break up libextra in order to foster futher innovation of crates. Development times will improve (smaller crates), more functionality can come in (not a giant crate to worry about), and existing functionality can be refined more quickly (smaller focus when dealing with a single nugget of functionality). I may be leaving something out, but that's the general idea in my mind at least. |
|
Something to keep in mind is that generic functions/containers don't slow down noticeably compile-time if they aren't used. The compile-time depends on the number of instantiations, which is going to go up if there are many crates reusing the same generic code. |
Also, crates that the compiler doesn't use will be exponentially improved, since they will not require a full bootstrap when changed. |
|
before this dissolution: after this dissolution: You see, more lines ( Before module system being improved (Issue #11745), I wouldn't like to see landing this PR personally. |
|
@liigo You're missing the point. |
|
I see your point, but i think it is worth mentioning that you will only need the extra |
|
Thank you so much for starting this. You are my hero. |
|
@ligo I think that should not be the case since promoting extra mods to crates will raise the nesting up a level, i.e instead of: You'll write In either case, afterward you are just using In submodules where you previously wrote |
|
My own perspective on why we're doing this: It's long been a consensus that smaller libraries are better for Rust's compilation model, and we have a strong desire to not have a large, monolithic, and in some ways inflexible standard library (like Java). So we want to have a strong ecosystem of easily accessible and featureful crates, some of which are officially supported and part of the 'standard rust distribution'. So the standard library is intended to be small and focused on core language and runtime infrastructure, but the standard libraries should be expansive. We're having a hard time getting to this stage because of the delays in finishing rustpkg. Rust gets a lot of pull requests for interesting library functionality, and we want to encourage that while also resisting piling everything into std, so as an interim solution to support the continued expansion of Rust's libraries we need to get libextra broken up so that the library organization begins to look like our our eventual goals, and we get experince with the many-crates model. |
|
Rust's design is oriented toward concerns of “programming in the large”. (from rust-lang.org) Is it right? After extra's dissolution: |
|
The conventional style is for extern mod extra;
use extra::json;
use extra::flate;
fn main() { flate::foo(); json::bar(); }Under this scheme extern mod json;
extern mod flate;
fn main() { flate::foo(); json::bar(); } |
|
@ligo Yes, Rust is concerned with 'programming in the large', and yes, your example is correct. Do you have a suggestion on how to reduce the amount of syntax here? |
|
@ligo I'd suggest that this problem is relatively minor at scale. Yes, for small projects where there are no submodules, you may have some extra |
|
Great diffstats on this: 544 additions and 1,039 deletions. |
|
@brson Yes, I have: make the When programmers write I've leave comments to #11745 yesterday. This is liigo not ligo, thank you. |
|
I like that this apparently replaces the |
|
brson: OK, I will. @huonw : for rust-style source code, you need |
|
Yes, |
|
mozilla/rust repo always |
|
No, it really doesn't. Random example: everything except |
|
It is minority exceptional case at all. |
|
Well, this is just amazing. It's impossible to guess whether this is correct or not since the makefiles are so blindingly impenetrable, so we're just going to have to push it through and see what happens, but this is a massive improvement to the design of the build system. |
|
@liigo I'm sorry, but it's not. Even the style guide says "Prefer to fully import types while module-qualifying functions" (my emphasis). Some basic statistics: the first is counting the number of occurrences of $ git grep 'use std::[^:]*;' | wc -l
936
$ git grep 'use std::[^:]*::' | wc -l
1096
$ git grep 'use std::[^:]*::[^A-Z]' | wc -l
650So, one certainly cannot say that using a module is rare. Also note that the second number includes imports like (In any case, this discussion is offtopic for this PR.) |
Before this patch, if you wanted to add a crate to the build system you had to change about 100 lines across 8 separate makefiles. This is highly error prone and opaque to all but a few. This refactoring is targeted at consolidating this effort so adding a new crate adds one line in one file in a way that everyone can understand it.
This is hopefully the beginning of the long-awaited dissolution of libextra. Using the newly created build infrastructure for building libraries, I decided to move the first module out of libextra. While not being a particularly meaty module in and of itself, the flate module is required by rustc and additionally has a native C dependency. I was able to very easily split out the C dependency from rustrt, update librustc, and magically everything gets installed to the right locations and built automatically. This is meant to be a proof-of-concept commit to how easy it is to remove modules from libextra now. I didn't put any effort into modernizing the interface of libflate or updating it other than to remove the one glob import it had.
It was decided a long, long time ago that libextra should not exist, but rather its modules should be split out into smaller independent libraries maintained outside of the compiler itself. The theory was to use `rustpkg` to manage dependencies in order to move everything out of the compiler, but maintain an ease of usability. Sadly, the work on `rustpkg` isn't making progress as quickly as expected, but the need for dissolving libextra is becoming more and more pressing. Because of this, we've thought that a good interim solution would be to simply package more libraries with the rust distribution itself. Instead of dissolving libextra into libraries outside of the mozilla/rust repo, we can dissolve libraries into the mozilla/rust repo for now. Work on this has been excruciatingly painful in the past because the makefiles are completely opaque to all but a few. Adding a new library involved adding about 100 lines spread out across 8 files (incredibly error prone). The first commit of this pull request targets this pain point. It does not rewrite the build system, but rather refactors large portions of it. Afterwards, adding a new library is as simple as modifying 2 lines (easy, right?). The build system automatically keeps track of dependencies between crates (rust *and* native), promotes binaries between stages, tracks dependencies of installed tools, etc, etc. With this newfound buildsystem power, I chose the `extra::flate` module as the first candidate for removal from libextra. While a small module, this module is relative complex in that is has a C dependency and the compiler requires it (messing with the dependency graph a bit). Albeit I modified more than 2 lines of makefiles to accomodate libflate (the native dependency required 2 extra lines of modifications), but the removal process was easy to do and straightforward. --- Testing-wise, I've cross-compiled, run tests, built some docs, installed, uninstalled, etc. I'm still working out a few kinks, and I'm sure that there's gonna be built system issues after this, but it should be working well for basic use! cc #8784
In line with the dissolution of libextra - rust-lang#8784 - moves arena to its own library libarena. Changes based on PR rust-lang#11787. Updates .gitignore to ignore doc/arena.
In line with the dissolution of libextra - moves glob to its own library libglob. Changes based on PR rust-lang#11787. Updates .gitignore to ignore doc/glob.
Fixes to `manual_let_else`'s divergence check
A few changes to the divergence check in `manual_let_else` and moves it the implementation to `clippy_utils` since it's generally useful:
* Handle internal `break` and `continue` expressions.
e.g. The first loop is divergent, but the second is not.
```rust
{
loop {
break 'outer;
};
}
{
loop {
break;
};
}
```
* Match rust's definition of divergence which is defined via the type system.
e.g. The following is not considered divergent by rustc as the inner block has a result type of `()`:
```rust
{
'a: {
panic!();
break 'a;
};
}
```
* Handle when adding a single semicolon would make the expression divergent.
e.g. The following would be a divergent if a semicolon were added after the `if` expression:
```rust
{ if panic!() { 0 } else { 1 } }
```
changelog: None
It was decided a long, long time ago that libextra should not exist, but rather its modules should be split out into smaller independent libraries maintained outside of the compiler itself. The theory was to use
rustpkgto manage dependencies in order to move everything out of the compiler, but maintain an ease of usability.Sadly, the work on
rustpkgisn't making progress as quickly as expected, but the need for dissolving libextra is becoming more and more pressing. Because of this, we've thought that a good interim solution would be to simply package more libraries with the rust distribution itself. Instead of dissolving libextra into libraries outside of the mozilla/rust repo, we can dissolve libraries into the mozilla/rust repo for now.Work on this has been excruciatingly painful in the past because the makefiles are completely opaque to all but a few. Adding a new library involved adding about 100 lines spread out across 8 files (incredibly error prone). The first commit of this pull request targets this pain point. It does not rewrite the build system, but rather refactors large portions of it. Afterwards, adding a new library is as simple as modifying 2 lines (easy, right?). The build system automatically keeps track of dependencies between crates (rust and native), promotes binaries between stages, tracks dependencies of installed tools, etc, etc.
With this newfound buildsystem power, I chose the
extra::flatemodule as the first candidate for removal from libextra. While a small module, this module is relative complex in that is has a C dependency and the compiler requires it (messing with the dependency graph a bit). Albeit I modified more than 2 lines of makefiles to accomodate libflate (the native dependency required 2 extra lines of modifications), but the removal process was easy to do and straightforward.Testing-wise, I've cross-compiled, run tests, built some docs, installed, uninstalled, etc. I'm still working out a few kinks, and I'm sure that there's gonna be built system issues after this, but it should be working well for basic use!
cc #8784