This guide is meant to help document how rustc – the Rust compiler –
+works, as well as to help new contributors get involved in rustc
+development.
+The Guide itself is of course open-source as well, and the sources can
+be found at the GitHub repository. If you find any mistakes in the
+guide, please file an issue about it, or even better, open a PR
+with a correction!
+This section of the rustc-dev-guide contains knowledge that should be useful to you
+regardless of what part of the compiler you are working on. This includes both
+technical info and tips (e.g. how to compile and debug the compiler) and info
+about processes in the Rust project (e.g. stabilization and info about the
+compiler team).
+If you're interested in figuring out who can answer questions about a
+particular part of the compiler, or you'd just like to know who works on what,
+check out our experts directory.
+It contains a listing of the various parts of the compiler and a list of people
+who are experts on each one.
+The compiler team has a weekly meeting where we do triage and try to
+generally stay on top of new bugs, regressions, and other things.
+They are held on Zulip. It works roughly as follows:
+The meeting currently takes place on Thursdays at 10am Boston time
+(UTC-4 typically, but daylight savings time sometimes makes things
+complicated).
+Membership in the Rust team is typically offered when someone has been
+making significant contributions to the compiler for some
+time. Membership is both a recognition but also an obligation:
+compiler team members are generally expected to help with upkeep as
+well as doing reviews and other work.
+If you are interested in becoming a compiler team member, the first
+thing to do is to start fixing some bugs, or get involved in a working
+group. One good way to find bugs is to look for
+open issues tagged with E-easy
+or
+E-mentor.
+Once you have made a number of individual PRs to rustc, we will often
+offer r+ privileges. This means that you have the right to instruct
+"bors" (the robot that manages which PRs get landed into rustc) to
+merge a PR
+(here are some instructions for how to talk to bors).
+Getting on the high-five list is much appreciated as it lowers the
+review burden for all of us! However, if you don't have time to give
+people timely feedback on their PRs, it may be better that you don't
+get on the list.
+Full team membership is typically extended once someone made many
+contributions to the Rust compiler over time, ideally (but not
+necessarily) to multiple areas. Sometimes this might be implementing a
+new feature, but it is also important — perhaps more important! — to
+have time and willingness to help out with general upkeep such as
+bugfixes, tracking regressions, and other less glamorous work.
+Then you will want to open up the file and change the following
+settings (and possibly others, such as llvm.ccache):
+This chapter focuses on the basics to be productive, but
+if you want to learn more about x.py, read its README.md
+here.
+For hacking, often building the stage 1 compiler is enough, but for
+final testing and release, the stage 2 compiler is used.
+This final product (stage1 compiler + libs built using that compiler)
+is what you need to build other rust programs (unless you use #![no_std] or
+#![no_core]).
+Unfortunately, incremental cannot be used to speed up making the
+stage1 libraries. This is because incremental only works when you run
+the same compiler twice in a row. In this case, we are building a
+new stage1 compiler every time. Therefore, the old incremental
+results may not apply. As a result, you will probably find that
+building the stage1 libstd is a bottleneck for you -- but fear not,
+there is a (hacky) workaround. See the section on "recommended
+workflows" below.
+Sometimes you might just want to test if the part you’re working on can
+compile. Using these commands you can test that it compiles before doing
+a bigger build to make sure it works with the compiler. As shown before
+you can also pass flags at the end such as --stage.
+Sometimes you need to start fresh, but this is normally not the case.
+If you need to run this then rustbuild is most likely not acting right and
+you should file a bug as to what is going wrong. If you do need to clean
+everything up then you only need to run one command!
+The full bootstrapping process takes quite a while. Here are three suggestions
+to make your life easier.
+In fact, it is sometimes useful to put off tests even when you are not
+100% sure the code will work. You can then keep building up
+refactoring commits and only run the tests at some later time. You can
+then use git bisect to track down precisely which commit caused
+the problem. A nice side-effect of this style is that you are left
+with a fairly fine-grained set of commits at the end, all of which
+build and pass tests. This often helps reviewing.
+Sometimes just checking
+whether the compiler builds is not enough. A common example is that
+you need to add a debug! statement to inspect the value of some
+state or better understand the problem. In that case, you really need
+a full build. By leveraging incremental, though, you can often get
+these builds to complete very fast (e.g., around 30 seconds). The only
+catch is this requires a bit of fudging and may produce compilers that
+don't work (but that is easily detected and fixed).
+By default, LLVM is built from source, and that can take significant amount of
+time. An alternative is to use LLVM already installed on your computer.
+We have observed the following paths before, which may be different from your system:
+This subchapter is about the bootstrapping process.
+Keep in mind this diagram is a simplification, i.e. rustdoc can be built at
+different stages, the process is a bit different when passing flags such as
+--keep-stage, or if there are non-host targets.
+So "stage0 std artifacts" are in fact the output of the downloaded stage0
+compiler, and are going to be used for anything built by the stage0 compiler:
+e.g. rustc artifacts. When it announces that it is "building stage1
+std artifacts" it has moved on to the next bootstrapping phase. This pattern
+continues in latter stages.
+The reason we need to build it twice is because of ABI compatibility.
+The beta compiler has it's own ABI, and then the stage1/bin/rustc compiler
+will produce programs/libraries with the new ABI.
+We used to build three times, but because we assume that the ABI is constant
+within a codebase, we presume that the libraries produced by the "stage2"
+compiler (produced by the stage1/bin/rustc compiler) is ABI-compatible with
+the stage1/bin/rustc compiler's produced libraries.
+What this means is that we can skip that final compilation -- and simply use the
+same libraries as the stage2/bin/rustc compiler uses itself for programs it
+links against.
+During bootstrapping, there are a bunch of compiler-internal environment
+variables that are used. If you are trying to run an intermediate version of
+rustc, sometimes you may need to set some of these environment variables
+manually. Otherwise, you get an error like the following:
+You might want to build and package up the compiler for distribution.
+You’ll want to run this command to do it:
+If you’ve built a distribution artifact you might want to install it and
+test that it works on your target system. You’ll want to run this command:
+Note: If you are testing out a modification to a compiler, you
+might want to use it to compile some project.
+Usually, you do not want to use ./x.py install for testing.
+Rather, you should create a toolchain as discussed in
+here.
+For example, if the toolchain you created is called foo, you
+would then invoke it with rustc +foo ... (where ... represents
+the rest of the arguments).
+You might want to build documentation of the various components
+available like the standard library. There’s two ways to go about this.
+You can run rustdoc directly on the file to make sure the HTML is
+correct, which is fast. Alternatively, you can build the documentation
+as part of the build process through x.py. Both are viable methods
+since documentation is more about the content.
+First the compiler and rustdoc get built to make sure everything is okay
+and then it documents the files.
+Much like individual tests or building certain components you can build only
+the documentation you want.
+Compiler documentation is not built by default. To enable it, modify config.toml:
+Note that when enabled,
+documentation for internal compiler items will also be built.
+One of the challenges with rustc is that the RLS can't handle it, since it's a
+bootstrapping compiler. This makes code navigation difficult. One solution is to
+use ctags.
+This allows you to do "jump-to-def" with whatever functions were around when
+you last built, which is ridiculously useful.
+The Rust project runs a wide variety of different tests, orchestrated by
+the build system (x.py test). The main test harness for testing the
+compiler itself is a tool called compiletest (located in the
+src/tools/compiletest directory). This section gives a brief
+overview of how the testing framework is setup, and then gets into some
+of the details on how to run tests as well as how to
+add new tests.
+Here is a brief summary of the test suites and what they mean. In some
+cases, the test suites are linked to parts of the manual that give more
+details.
+The Rust build system handles running tests for various other things,
+including:
+Tests may be run on a remote machine (e.g. to test builds for a different
+architecture). This is done using remote-test-client on the build machine
+to send test programs to remote-test-server running on the remote machine.
+remote-test-server executes the test programs and sends the results back to
+the build machine. remote-test-server provides unauthenticated remote code
+execution so be careful where it is used.
+Some platforms are tested via an emulator for architectures that aren't
+readily available. For architectures where the standard library is well
+supported and the host operating system supports TCP/IP networking, see the
+above instructions for testing on a remote machine (in this case the
+remote machine is emulated).
+There is also a set of tools for orchestrating running the
+tests within the emulator. Platforms such as arm-android and
+arm-unknown-linux-gnueabihf are set up to automatically run the tests under
+emulation on Travis. The following will take a look at how a target's tests
+are run under emulation.
+You should request a crater run if your PR makes large changes to the compiler
+or could cause breakage. If you are unsure, feel free to ask your PR's reviewer.
+The rust team maintains a few machines that can be used for running crater runs
+on the changes introduced by a PR. If your PR needs a crater run, leave a
+comment for the triage team in the PR thread. Please inform the team whether
+you require a "check-only" crater run, a "build only" crater run, or a
+"build-and-test" crater run. The difference is primarily in time; the
+conservative (if you're not sure) option is to go for the build-and-test run.
+If making changes that will only have an effect at compile-time (e.g.,
+implementing a new trait) then you only need a check run.
+Your PR will be enqueued by the triage team and the results will be posted when
+they are ready. Check runs will take around ~3-4 days, with the other two
+taking 5-6 days on average.
+While crater is really useful, it is also important to be aware of a few
+caveats:
+A lot of work is put into improving the performance of the compiler and
+preventing performance regressions. A "perf run" is used to compare the
+performance of the compiler in different configurations for a large collection
+of popular crates. Different configurations include "fresh builds", builds
+with incremental compilation, etc.
+The result of a perf run is a comparison between two versions of the
+compiler (by their commit hashes).
+You should request a perf run if your PR may affect performance, especially
+if it can affect performance adversely.
+This will build the full stage 2 compiler and then run the whole test
+suite. You probably don't want to do this very often, because it takes
+a very long time, and anyway bors / travis will do it for you. (Often,
+I will run this command in the background after opening a PR that I
+think is done, but rarely otherwise. -nmatsakis)
+Note that some tests require a Python-enabled gdb. You can test if
+your gdb install supports Python by using the python command from
+within gdb. Once invoked you can type some Python code (e.g.
+print("hi")) followed by return and then CTRL+D to execute it.
+If you are building gdb from source, you will need to configure with
+--with-python=<path-to-python-binary>.
+When working on a specific PR, you will usually want to run a smaller
+set of tests, and with a stage 1 build. For example, a good "smoke
+test" that can be used after modifying rustc to see if things are
+generally working correctly would be the following:
+If you only need to test a specific subdirectory of tests for any
+given test suite, you can pass that directory to x.py test:
+By listing which test suites you want to run you avoid having to run
+tests for components you did not change at all.
+Under the hood, the test runner invokes the standard rust test runner
+(the same one you get with #[test]), so this command would wind up
+filtering for tests that include "issue-1234" in the name. (Thus
+--test-args is a good way to run a collection of related tests.)
+If you have changed the compiler's output intentionally, or you are
+making a new test, you can pass --bless to the test subcommand. E.g.
+if some tests in src/test/ui are failing, you can run
+If you don't want to include the flag with every command, you can
+enable it in the config.toml:
+Note that incremental compilation will use more disk space than usual.
+If disk space is a concern for you, you might want to check the size
+of the build directory from time to time.
+Sometimes it's easier and faster to just run the test by hand. Most tests are
+just rs files, so you can do something like
+This is much faster, but doesn't always work. For example, some tests
+include directives that specify specific compiler flags, or which rely
+on other crates, and they may not run the same without those options.
+To add a new test, the first thing you generally do is to create a
+file, typically a Rust source file. Test files have a particular
+structure:
+It can be difficult to know what kind of test to use. Here are some
+rough heuristics:
+We have not traditionally had a lot of structure in the names of
+tests. Moreover, for a long time, the rustc test runner did not
+support subdirectories (it now does), so test suites like
+src/test/ui have a huge mess of files in them. This is not
+considered an ideal setup.
+For regression tests – basically, some random snippet of code that
+came in from the internet – we often name the test after the issue
+plus a short description. Ideally, the test should be added to a
+directory that helps identify what piece of code is being tested here
+(e.g., src/test/ui/borrowck/issue-54597-reject-move-out-of-borrow-via-pat.rs)
+If you've tried and cannot find a more relevant place,
+the test may be added to src/test/ui/issues/.
+Still, do include the issue number somewhere.
+In other cases, there may already be a suitable directory. (The proper
+directory structure to use is actually an area of active debate.)
+This comment doesn't have to be super extensive. Just something like
+"Regression test for #18060: match arms were matching in the wrong
+order." might already be enough.
+These comments are very useful to others later on when your test
+breaks, since they often can highlight what the problem is. They are
+also useful if for some reason the tests need to be refactored, since
+they let others know which parts of the test were important (often a
+test must be rewritten because it no longer tests what is was meant to
+test, and then it's useful to know what it was meant to test
+exactly).
+Header commands are special comments that the test runner knows how to
+interpret. They must appear before the Rust source in the test. They
+are normally put after the short comment that explains the point of
+this test. For example, this test uses the // compile-flags command
+to specify a custom flag to give to rustc when the test is compiled:
+These are used to ignore the test in some situations, which means the test won't
+be compiled or run.
+Here is a list of other header commands. This list is not
+exhaustive. Header commands can generally be found by browsing the
+TestProps structure found in header.rs from the compiletest
+source.
+Error annotations specify the errors that the compiler is expected to
+emit. They are "attached" to the line in source where the error is
+located. Error annotations are considered during tidy lints of line
+length and should be formatted according to tidy requirements. You may
+use an error message prefix sub-string if necessary to meet line length
+requirements. Make sure that the text is long enough for the error
+message to be self-documenting.
+The error annotation definition and source line definition association
+is defined with the following set of idioms:
+Here are examples of error annotations on different lines of UI test
+source.
+Certain classes of tests support "revisions" (as of the time of this
+writing, this includes compile-fail, run-fail, and
+incremental, though incremental tests are somewhat
+different). Revisions allow a single test file to be used for multiple
+tests. This is done by adding a special header at the top of the file:
+This will result in the test being compiled (and tested) three times,
+once with --cfg foo, once with --cfg bar, and once with --cfg baz. You can therefore use #[cfg(foo)] etc within the test to tweak
+each of these results.
+You can also customize headers and expected error messages to a particular
+revision. To do this, add [foo] (or bar, baz, etc) after the //
+comment, like so:
+Note that not all headers have meaning when customized to a revision.
+For example, the ignore-test header (and all "ignore" headers)
+currently only apply to the test as a whole, not to particular
+revisions. The only headers that are intended to really work when
+customized to a revision are error patterns and compiler flags.
+The UI tests are intended to capture the compiler's complete output,
+so that we can test all aspects of the presentation. They work by
+compiling a file (e.g., ui/hello_world/main.rs),
+capturing the output, and then applying some normalization (see
+below). This normalized result is then compared against reference
+files named ui/hello_world/main.stderr and
+ui/hello_world/main.stdout. If either of those files doesn't exist,
+the output must be empty (that is actually the case for
+this particular test). If the test run fails, we will print out
+the current output, but it is also saved in
+build/<target-triple>/test/ui/hello_world/main.stdout (this path is
+printed as part of the test failure message), so you can run diff
+and so forth.
+The normalization applied is aimed at eliminating output difference
+between platforms, mainly about filenames:
+Sometimes these built-in normalizations are not enough. In such cases, you
+may provide custom normalization rules using the header commands, e.g.
+The corresponding reference file will use the normalized output to test both
+32-bit and 64-bit platforms:
+The tests themselves are typically (but not always) organized into
+"suites" – for example, run-fail,
+a folder holding tests that should compile successfully,
+but return a failure (non-zero status), compile-fail, a folder holding tests
+that should fail to compile, and many more. The various suites are defined in
+src/tools/compiletest/src/common.rs in the pub enum Mode
+declaration. And a very good introduction to the different suites of compiler
+tests along with details about them can be found in Adding new
+tests.
+Source file annotations which appear in comments near the top of the source
+file before any test code are known as header commands. These commands can
+instruct compiletest to ignore this test, set expectations on whether it is
+expected to succeed at compiling, or what the test's return code is expected to
+be. Header commands (and their inline counterparts, Error Info commands) are
+described more fully
+here.
+One would add a new header command if there is a need to define some test
+property or behavior on an individual, test-by-test basis. A header command
+property serves as the header command's backing store (holds the command's
+current value) at runtime.
+Parsers will override a given header command property's default value merely by
+being specified in the test file as a header command or by having a parameter
+value specified in the test file, depending on the header command.
+When a test invokes a particular header command, it is expected that some
+behavior will change as a result. What behavior, obviously, will depend on the
+purpose of the header command. In the case of failure-status, the behavior
+that changes is that compiletest expects the failure code defined by the
+header command invoked in the test, rather than the default value.
+In general, if you are interested in making a contribution and aren't sure
+where to start, please feel free to ask!
+There were a number of steps to go from an idea to stable rust feature. Here is
+a quick list. We will go through each of these in order below. As I mentioned
+before, not all of these are needed for every type of contribution.
+An RFC is a document that describes the feature or change you are proposing in
+detail. Anyone can write an RFC; the process is the same for everyone,
+including rust team members.
+Before opening an RFC, you should do the research to "flesh out" your idea.
+Hastily-proposed RFCs tend not to be accepted. You should generally have a good
+description of the motivation, impact, disadvantages, and potential
+interactions with other features.
+If that sounds like a lot of work, it's because it is. But no fear! Even if
+you're not a compiler hacker, you can get great feedback by doing a pre-RFC.
+This is an informal discussion of the idea. The best place to do this is
+internals.rust-lang.org. Your post doesn't have to follow any particular
+structure. It doesn't even need to be a cohesive idea. Generally, you will get
+tons of feedback that you can integrate back to produce a good RFC.
+(Another pro-tip: try searching the RFCs repo and internals for prior related
+ideas. A lot of times an idea has already been considered and was either
+rejected or postponed to be tried again later. This can save you and everybody
+else some time)
+In the case of our example, a participant in the pre-RFC thread pointed out a
+syntax ambiguity and a potential resolution. Also, the overall feedback seemed
+positive. In this case, the discussion converged pretty quickly, but for some
+ideas, a lot more discussion can happen (e.g. see this RFC which
+received a whopping 684 comments!). If that happens, don't be discouraged; it
+means the community is interested in your idea, but it perhaps needs some
+adjustments.
+In the end, when the discussion seems to reach a consensus and die down a bit,
+a rust team member may propose to move to "final comment period" (FCP) with one
+of three possible dispositions. This means that they want the other members of
+the appropriate teams to review and comment on the RFC. More discussion may
+ensue, which may result in more changes or unresolved questions being added. At
+some point, when everyone is satisfied, the RFC enters the FCP, which is the
+last chance for people to bring up objections. When the FCP is over, the
+disposition is adopted. Here are the three possible dispositions:
+Depending on the feature/change/bug fix/improvement, implementation may be
+relatively-straightforward or it may be a major undertaking. You can always ask
+for help or mentorship from more experienced compiler devs. Also, you don't
+have to be the one to implement your feature; but keep in mind that if you
+don't it might be a while before someone else does.
+The reviewer may request changes before they approve your PR. Feel free to ask
+questions or discuss things you don't understand or disagree with. However,
+recognize that the PR won't be merged unless someone on the rust team approves
+it.
+When your review approves the PR, it will go into a queue for yet another bot
+called @bors. @bors manages the CI build/merge queue. When your PR reaches
+the head of the @bors queue, @bors will test out the merge by running all
+tests against your PR on Travis CI. This takes a lot of time to
+finish. If all tests pass, the PR is merged and becomes part of the next
+nightly compiler!
+There are a couple of things that may happen for some PRs during the review process
+If you are not doing a new feature or something like that (e.g. if you are
+fixing a bug), then that's it! Thanks for your contribution :)
+As people get experience with your new feature on nightly, slight changes may
+be proposed and unresolved questions may become resolved. Updates/changes go
+through the same process for implementing any other changes, as described
+above (i.e. submit a PR, go through review, wait for @bors, etc).
+Some changes may be major enough to require an FCP and some review by rust team
+members.
+Finally, after the feature had baked for a while on nightly, a language team member
+moved to stabilize it.
+This page defines the best practices procedure for making bug fixes or soundness
+corrections in the compiler that can cause existing code to stop compiling. This
+text is based on
+RFC 1589.
+From time to time, we encounter the need to make a bug fix, soundness
+correction, or other change in the compiler which will cause existing code to
+stop compiling. When this happens, it is important that we handle the change in
+a way that gives users of Rust a smooth transition. What we want to avoid is
+that existing programs suddenly stop compiling with opaque error messages: we
+would prefer to have a gradual period of warnings, with clear guidance as to
+what the problem is, how to fix it, and why the change was made. This RFC
+describes the procedure that we have been developing for handling breaking
+changes that aims to achieve that kind of smooth transition.
+One of the key points of this policy is that (a) warnings should be issued
+initially rather than hard errors if at all possible and (b) every change that
+causes existing code to stop compiling will have an associated tracking issue.
+This issue provides a point to collect feedback on the results of that change.
+Sometimes changes have unexpectedly large consequences or there may be a way to
+avoid the change that was not considered. In those cases, we may decide to
+change course and roll back the change, or find another solution (if warnings
+are being used, this is particularly easy to do).
+Note that this RFC does not try to define when a breaking change is permitted.
+That is already covered under RFC 1122. This document assumes that the
+change being made is in accordance with those policies. Here is a summary of the
+conditions from RFC 1122:
+The procedure for making a breaking change is as follows (each of these steps is
+described in more detail below):
+A template for these breaking-change tracking issues can be found below. An
+example of how such an issue should look can be found
+here.
+The best way to handle a breaking change is to begin by issuing
+future-compatibility warnings. These are a special category of lint warning.
+Adding a new future-compatibility warning can be done as follows.
+It can often be challenging to filter out new warnings from older, pre-existing
+errors. One technique that has been used in the past is to run the older code
+unchanged and collect the errors it would have reported. You can then issue
+warnings for any errors you would give which do not appear in that original set.
+Another option is to abort compilation after the original code completes if
+errors are reported: then you know that your new code will only execute when
+there were no errors before.
+We should always do a crater run to assess impact. It is polite and considerate
+to at least notify the authors of affected crates the breaking change. If we can
+submit PRs to fix the problem, so much the better.
+Changes that are believed to have negligible impact can go directly to issuing
+an error. One rule of thumb would be to check against crates.io: if fewer than
+10 total affected projects are found (not root errors), we can move
+straight to an error. In such cases, we should still make the "breaking change"
+page as before, and we should ensure that the error directs users to this page.
+In other words, everything should be the same except that users are getting an
+error, and not a warning. Moreover, we should submit PRs to the affected
+projects (ideally before the PR implementing the change lands in rustc).
+If the impact is not believed to be negligible (e.g., more than 10 crates are
+affected), then warnings are required (unless the compiler team agrees to grant
+a special exemption in some particular case). If implementing warnings is not
+feasible, then we should make an aggressive strategy of migrating crates before
+we land the change so as to lower the number of affected crates. Here are some
+techniques for approaching this scenario:
+
+
+Once we have decided to make a "future warning" into a hard error, we need a PR
+that removes the custom lint. As an example, here are the steps required to
+remove the overlapping_inherent_impls compatibility lint. First, convert the
+name of the lint to uppercase (OVERLAPPING_INHERENT_IMPLS) ripgrep through the
+source for that string. We will basically by converting each place where this
+lint name is mentioned (in the compiler, we use the upper-case name, and a macro
+automatically generates the lower-case string; so searching for
+overlapping_inherent_impls would not find much).
+
+NOTE: these exact files don't exist anymore, but the procedure is still the same.
+
+
+The first reference you will likely find is the lint definition in
+librustc/lint/builtin.rs that resembles this:
+
+#![allow(unused_variables)]
+fn main() {
+declare_lint! {
+ pub OVERLAPPING_INHERENT_IMPLS,
+ Deny, // this may also say Warning
+ "two overlapping inherent impls define an item with the same name were erroneously allowed"
+}
+}
+
+This declare_lint! macro creates the relevant data structures. Remove it. You
+will also find that there is a mention of OVERLAPPING_INHERENT_IMPLS later in
+the file as part of a lint_array!; remove it too,
+Next, you see see a reference to OVERLAPPING_INHERENT_IMPLS in
+librustc_lint/lib.rs. This defining the lint as a "future
+compatibility lint":
+
+#![allow(unused_variables)]
+fn main() {
+FutureIncompatibleInfo {
+ id: LintId::of(OVERLAPPING_INHERENT_IMPLS),
+ reference: "issue #36889 <https://github.com/rust-lang/rust/issues/36889>",
+},
+}
+
+Remove this too.
+
+In src/librustc_lint/lib.rs there is a list of "renamed and removed lints".
+You can add this lint to the list:
+
+#![allow(unused_variables)]
+fn main() {
+store.register_removed("overlapping_inherent_impls", "converted into hard error, see #36889");
+}
+
+where #36889 is the tracking issue for your lint.
+
+Finally, the last class of references you will see are the places that actually
+trigger the lint itself (i.e., what causes the warnings to appear). These
+you do not want to delete. Instead, you want to convert them into errors. In
+this case, the add_lint call looks like this:
+
+#![allow(unused_variables)]
+fn main() {
+self.tcx.sess.add_lint(lint::builtin::OVERLAPPING_INHERENT_IMPLS,
+ node_id,
+ self.tcx.span_of_impl(item1).unwrap(),
+ msg);
+}
+
+We want to convert this into an error. In some cases, there may be an
+existing error for this scenario. In others, we will need to allocate a
+fresh diagnostic code. Instructions for allocating a fresh diagnostic
+code can be found here. You may want
+to mention in the extended description that the compiler behavior
+changed on this point, and include a reference to the tracking issue for
+the change.
+Let's say that we've adopted E0592 as our code. Then we can change the
+add_lint() call above to something like:
+
+#![allow(unused_variables)]
+fn main() {
+struct_span_err!(self.tcx.sess, self.tcx.span_of_impl(item1).unwrap(), msg)
+ .emit();
+}
+
+
+Finally, run the test suite. These should be some tests that used to reference
+the overlapping_inherent_impls lint, those will need to be updated. In
+general, if the test used to have #[deny(overlapping_inherent_impls)], that
+can just be removed.
+./x.py test
+
+
+Open a PR. =)
+
+
+When you want to implement a new significant feature in the compiler,
+you need to go through this process to make sure everything goes
+smoothly.
+
+When the change is small and uncontroversial, then it can be done
+with just writing a PR and getting r+ from someone who knows that
+part of the code. However, if the change is potentially controversial,
+it would be a bad idea to push it without consensus from the rest
+of the team (both in the "distributed system" sense to make sure
+you don't break anything you don't know about, and in the social
+sense to avoid PR fights).
+If such a change seems to be too small to require a full formal RFC
+process (e.g. a big refactoring of the code, or a
+"technically-breaking" change, or a "big bugfix" that basically
+amounts to a small feature) but is still too controversial or
+big to get by with a single r+, you can start a pFCP (or, if you
+don't have r+ rights, ask someone who has them to start one - and
+unless they have a concern themselves, they should).
+Again, the pFCP process is only needed if you need consensus - if you
+don't think anyone would have a problem with your change, it's ok to
+get by with only an r+. For example, it is OK to add or modify
+unstable command-line flags or attributes without an pFCP for
+compiler development or standard library use, as long as you don't
+expect them to be in wide use in the nightly ecosystem.
+You don't need to have the implementation fully ready for r+ to ask
+for a pFCP, but it is generally a good idea to have at least a proof
+of concept so that people can see what you are talking about.
+That starts a "proposed final comment period" (pFCP), which requires
+all members of the team to sign off the FCP. After they all do so,
+there's a 10 day long "final comment period" where everybody can comment,
+and if no new concerns are raised, the PR/issue gets FCP approval.
+
+There are a few "logistic" hoops you might need to go through in
+order to implement a feature in a working way.
+
+In some cases, a feature or bugfix might break some existing programs
+in some edge cases. In that case, you might want to do a crater run
+to assess the impact and possibly add a future-compatibility lint,
+similar to those used for
+edition-gated lints.
+
+We value the stability of Rust. Code that works and runs on stable
+should (mostly) not break. Because of that, we don't want to release
+a feature to the world with only team consensus and code review -
+we want to gain real-world experience on using that feature on nightly,
+and we might want to change the feature based on that experience.
+To allow for that, we must make sure users don't accidentally depend
+on that new feature - otherwise, especially if experimentation takes
+time or is delayed and the feature takes the trains to stable,
+it would end up de facto stable and we'll not be able to make changes
+in it without breaking people's code.
+The way we do that is that we make sure all new features are feature
+gated - they can't be used without enabling a feature gate
+(#[feature(foo)]), which can't be done in a stable/beta compiler.
+See the stability in code section for the technical details.
+Eventually, after we gain enough experience using the feature,
+make the necessary changes, and are satisfied, we expose it to
+the world using the stabilization process described here.
+Until then, the feature is not set in stone: every part of the
+feature can be changed, or the feature might be completely
+rewritten or removed. Features are not supposed to gain tenure
+by being unstable and unchanged for a year.
+
+
+To keep track of the status of an unstable feature, the
+experience we get while using it on nightly, and of the
+concerns that block its stabilization, every feature-gate
+needs a tracking issue.
+General discussions about the feature should be done on
+the tracking issue.
+For features that have an RFC, you should use the RFC's
+tracking issue for the feature.
+For other features, you'll have to make a tracking issue
+for that feature. The issue title should be "Tracking issue
+for YOUR FEATURE".
+For tracking issues for features (as opposed to future-compat
+warnings), I don't think the description has to contain
+anything specific. Generally we put the list of items required
+for stabilization in a checklist, e.g.,
+**Steps:**
+
+- [ ] Implement the RFC. (CC @rust-lang/compiler -- can anyone write
+ up mentoring instructions?)
+- [ ] Adjust the documentation. ([See instructions on rustc-dev-guide.](https://rustc-dev-guide.rust-lang.org/stabilization_guide.html#documentation-prs))
+- [ ] Stabilize the feature. ([See instructions on rustc-dev-guide.](https://rustc-dev-guide.rust-lang.org/stabilization_guide.html#stabilization-pr))
+
+
+
+The below steps needs to be followed in order to implement
+a new unstable feature:
+
+-
+
Open a tracking issue -
+if you have an RFC, you can use the tracking issue for the RFC.
+The tracking issue should be labeled with at least C-tracking-issue.
+For a language feature, a label F-feature_name should be added as well.
+
+-
+
Pick a name for the feature gate (for RFCs, use the name
+in the RFC).
+
+-
+
Add a feature gate declaration to librustc_feature/active.rs
+in the active declare_features block:
+/// description of feature
+(active, $feature_name, "$current_nightly_version", Some($tracking_issue_number), $edition)
+
+where $edition has the type Option<Edition>, and is typically
+just None.
+For example:
+/// Allows defining identifiers beyond ASCII.
+(active, non_ascii_idents, "1.0.0", Some(55467), None),
+
+When added, the current version should be the one for the current nightly.
+Once the feature is moved to accepted.rs, the version is changed to that nightly version.
+
+-
+
Prevent usage of the new feature unless the feature gate is set.
+You can check it in most places in the compiler using the
+expression tcx.features().$feature_name (or
+sess.features_untracked().$feature_name if the
+tcx is unavailable)
+If the feature gate is not set, you should either maintain
+the pre-feature behavior or raise an error, depending on
+what makes sense.
+For features introducing new syntax, pre-expansion gating should be used instead.
+To do so, extend the GatedSpans struct, add spans to it during parsing,
+and then finally feature-gate all the spans in
+rustc_ast_passes::feature_gate::check_crate.
+
+-
+
Add a test to ensure the feature cannot be used without
+a feature gate, by creating feature-gate-$feature_name.rs
+and feature-gate-$feature_name.stderr files under the
+directory where the other tests for your feature reside.
+
+-
+
Add a section to the unstable book, in
+src/doc/unstable-book/src/language-features/$feature_name.md.
+
+-
+
Write a lots of tests for the new feature.
+PRs without tests will not be accepted!
+
+-
+
Get your PR reviewed and land it. You have now successfully
+implemented a feature in Rust!
+
+
+
+This section is about the stability attributes and schemes that allow stable
+APIs to use unstable APIs internally in the rustc standard library.
+For instructions on stabilizing a language feature see Stabilizing
+Features.
+
+The #[unstable(feature = "foo", issue = "1234", reason = "lorem ipsum")]
+attribute explicitly marks an item as unstable. Items that are marked as
+"unstable" cannot be used without a corresponding #![feature] attribute on
+the crate, even on a nightly compiler. This restriction only applies across
+crate boundaries, unstable items may be used within the crate that defines
+them.
+The issue field specifies the associated GitHub issue number. This field is
+required and all unstable features should have an associated tracking issue. In
+rare cases where there is no sensible value issue = "none" is used.
+The unstable attribute infects all sub-items, where the attribute doesn't
+have to be reapplied. So if you apply this to a module, all items in the module
+will be unstable.
+You can make specific sub-items stable by using the #[stable] attribute on
+them. The stability scheme works similarly to how pub works. You can have
+public functions of nonpublic modules and you can have stable functions in
+unstable modules or vice versa.
+Note, however, that due to a rustc bug, stable items inside unstable modules
+are available to stable code in that location! So, for example, stable code
+can import core::intrinsics::transmute even though intrinsics is an
+unstable module. Thus, this kind of nesting should be avoided when possible.
+The unstable attribute may also have the soft value, which makes it a
+future-incompatible deny-by-default lint instead of a hard error. This is used
+by the bench attribute which was accidentally accepted in the past. This
+prevents breaking dependencies by leveraging Cargo's lint capping.
+
+The #[stable(feature = "foo", "since = "1.420.69")] attribute explicitly
+marks an item as stabilized. To do this, follow the instructions in
+Stabilizing Features.
+Note that stable functions may use unstable things in their body.
+
+The #[rustc_const_unstable(feature = "foo", issue = "1234", reason = "lorem ipsum")]
+has the same interface as the unstable attribute. It is used to mark
+const fn as having their constness be unstable. This allows you to make a
+function stable without stabilizing its constness or even just marking an existing
+stable function as const fn without instantly stabilizing the const fnness.
+Furthermore this attribute is needed to mark an intrinsic as const fn, because
+there's no way to add const to functions in extern blocks for now.
+
+The #[stable(feature = "foo", "since = "1.420.69")] attribute explicitly marks
+a const fn as having its constness be stable. This attribute can make sense
+even on an unstable function, if that function is called from another
+rustc_const_stable function.
+Furthermore this attribute is needed to mark an intrinsic as callable from
+rustc_const_stable functions.
+
+Macros, compiler desugarings and const fns expose their bodies to the call
+site. To work around not being able to use unstable things in the standard
+library's macros, there's the #[allow_internal_unstable(feature1, feature2)]
+attribute that whitelists the given features for usage in stable macros or
+const fns.
+Note that const fns are even more special in this regard. You can't just
+whitelist any feature, the features need an implementation in
+qualify_min_const_fn.rs. For example the const_fn_union feature gate allows
+accessing fields of unions inside stable const fns. The rules for when it's
+ok to use such a feature gate are that behavior matches the runtime behavior of
+the same code (see also this blog post). This means that you may not
+create a const fn that e.g. transmutes a memory address to an integer,
+because the addresses of things are nondeterministic and often unknown at
+compile-time.
+Always ping @oli-obk, @RalfJung, and @Centril if you are adding more
+allow_internal_unstable attributes to any const fn
+
+Any crate that uses the stable, unstable, or rustc_deprecated attributes
+must include the #![feature(staged_api)] attribute on the crate.
+
+The deprecation system shares the same infrastructure as the stable/unstable
+attributes. The rustc_deprecated attribute is similar to the deprecated
+attribute. It was previously called deprecated, but was split off when
+deprecated was stabilized. The deprecated attribute cannot be used in a
+staged_api crate, rustc_deprecated must be used instead. The deprecated
+item must also have a stable or unstable attribute.
+rustc_deprecated has the following form:
+#[rustc_deprecated(
+ since = "1.38.0",
+ reason = "explanation for deprecation",
+ suggestion = "other_function"
+)]
+
+The suggestion field is optional. If given, it should be a string that can be
+used as a machine-applicable suggestion to correct the warning. This is
+typically used when the identifier is renamed, but no other significant changes
+are necessary.
+Another difference from the deprecated attribute is that the since field is
+actually checked against the current version of rustc. If since is in a
+future version, then the deprecated_in_future lint is triggered which is
+default allow, but most of the standard library raises it to a warning with
+#![warn(deprecated_in_future)].
+
+The -Zforce-unstable-if-unmarked flag has a variety of purposes to help
+enforce that the correct crates are marked as unstable. It was introduced
+primarily to allow rustc and the standard library to link to arbitrary crates
+on crates.io which do not themselves use staged_api. rustc also relies on
+this flag to mark all of its crates as unstable with the rustc_private
+feature so that each crate does not need to be carefully marked with
+unstable.
+This flag is automatically applied to all of rustc and the standard library
+by the bootstrap scripts. This is needed because the compiler and all of its
+dependencies are shipped in the sysroot to all users.
+This flag has the following effects:
+
+- Marks the crate as "unstable" with the
rustc_private feature if it is not
+itself marked as stable or unstable.
+- Allows these crates to access other forced-unstable crates without any need
+for attributes. Normally a crate would need a
#![feature(rustc_private)]
+attribute to use other unstable crates. However, that would make it
+impossible for a crate from crates.io to access its own dependencies since
+that crate won't have a feature(rustc_private) attribute, but everything
+is compiled with -Zforce-unstable-if-unmarked.
+
+Code which does not use -Zforce-unstable-if-unmarked should include the
+#![feature(rustc_private)] crate attribute to access these force-unstable
+crates. This is needed for things that link rustc, such as miri, rls, or
+clippy.
+
+Once an unstable feature has been well-tested with no outstanding
+concern, anyone may push for its stabilization. It involves the
+following steps.
+
+- Documentation PRs
+- Write a stabilization report
+- FCP
+- Stabilization PR
+
+
+
+If any documentation for this feature exists, it should be
+in the Unstable Book, located at src/doc/unstable-book.
+If it exists, the page for the feature gate should be removed.
+If there was documentation there, integrating it into the
+existing documentation is needed.
+If there wasn't documentation there, it needs to be added.
+Places that may need updated documentation:
+
+- The Reference: This must be updated, in full detail.
+- The Book: This may or may not need updating, depends.
+If you're not sure, please open an issue on this repository
+and it can be discussed.
+- standard library documentation: As needed. Language features
+often don't need this, but if it's a feature that changes
+how good examples are written, such as when
? was added
+to the language, updating examples is important.
+- Rust by Example: As needed.
+
+Prepare PRs to update documentation involving this new feature
+for repositories mentioned above. Maintainers of these repositories
+will keep these PRs open until the whole stabilization process
+has completed. Meanwhile, we can proceed to the next step.
+
+Find the tracking issue of the feature, and create a short
+stabilization report. Essentially this would be a brief summary
+of the feature plus some links to test cases showing it works
+as expected, along with a list of edge cases that came up
+and were considered. This is a minimal "due diligence" that
+we do before stabilizing.
+The report should contain:
+
+- A summary, showing examples (e.g. code snippets) what is
+enabled by this feature.
+- Links to test cases in our test suite regarding this feature
+and describe the feature's behavior on encountering edge cases.
+- Links to the documentations (the PRs we have made in the
+previous steps).
+- Any other relevant information(Examples of such reports can
+be found in rust-lang/rust#44494 and rust-lang/rust#28237).
+- The resolutions of any unresolved questions if the stabilization
+is for an RFC.
+
+
+If any member of the team responsible for tracking this
+feature agrees with stabilizing this feature, they will
+start the FCP (final-comment-period) process by commenting
+@rfcbot fcp merge
+
+The rest of the team members will review the proposal. If the final
+decision is to stabilize, we proceed to do the actual code modification.
+
+Once we have decided to stabilize a feature, we need to have
+a PR that actually makes that stabilization happen. These kinds
+of PRs are a great way to get involved in Rust, as they take
+you on a little tour through the source code.
+Here is a general guide to how to stabilize a feature --
+every feature is different, of course, so some features may
+require steps beyond what this guide talks about.
+Note: Before we stabilize any feature, it's the rule that it
+should appear in the documentation.
+
+There is a central listing of feature-gates in
+src/librustc_feature. Search for the declare_features!
+macro. There should be an entry for the feature you are aiming
+to stabilize, something like (this example is taken from
+rust-lang/rust#32409:
+// pub(restricted) visibilities (RFC 1422)
+(active, pub_restricted, "1.9.0", Some(32409)),
+
+The above line should be moved down to the area for "accepted"
+features, declared below in a separate call to declare_features!.
+When it is done, it should look like:
+// pub(restricted) visibilities (RFC 1422)
+(accepted, pub_restricted, "1.31.0", Some(32409)),
+// note that we changed this
+
+Note that, the version number is updated to be the version number
+of the stable release where this feature will appear. This can be
+found by consulting the forge, which will guide
+you the next stable release number. You want to add 1 to that,
+because the code that lands today will become go into beta on that
+date, and then become stable after that. So, at the time of this
+writing, the next stable release (i.e. what is currently beta) was
+1.30.0, hence I wrote 1.31.0 above.
+
+Next search for the feature string (in this case, pub_restricted)
+in the codebase to find where it appears. Change uses of
+#![feature(XXX)] from the libstd and any rustc crates to be
+#![cfg_attr(bootstrap, feature(XXX))]. This includes the feature-gate
+only for stage0, which is built using the current beta (this is
+needed because the feature is still unstable in the current beta).
+Also, remove those strings from any tests. If there are tests
+specifically targeting the feature-gate (i.e., testing that the
+feature-gate is required to use the feature, but nothing else),
+simply remove the test.
+
+Most importantly, remove the code which flags an error if the
+feature-gate is not present (since the feature is now considered
+stable). If the feature can be detected because it employs some
+new syntax, then a common place for that code to be is in the
+same src/librustc_ast_passes/feature_gate.rs.
+For example, you might see code like this:
+gate_feature_post!(&self, pub_restricted, span,
+ "`pub(restricted)` syntax is experimental");
+
+This gate_feature_post! macro prints an error if the
+pub_restricted feature is not enabled. It is not needed
+now that #[pub_restricted] is stable.
+For more subtle features, you may find code like this:
+if self.tcx.sess.features.borrow().pub_restricted { /* XXX */ }
+
+This pub_restricted field (obviously named after the feature)
+would ordinarily be false if the feature flag is not present
+and true if it is. So transform the code to assume that the field
+is true. In this case, that would mean removing the if and
+leaving just the /* XXX */.
+if self.tcx.sess.features.borrow().pub_restricted { /* XXX */ }
+becomes
+/* XXX */
+
+if self.tcx.sess.features.borrow().pub_restricted && something { /* XXX */ }
+ becomes
+if something { /* XXX */ }
+
+
+This chapter contains a few tips to debug the compiler. These tips aim to be
+useful no matter what you are working on. Some of the other chapters have
+advice about specific parts of the compiler (e.g. the Queries Debugging and
+Testing chapter or the LLVM Debugging
+chapter).
+
+The compiler has a bunch of -Z flags. These are unstable flags that are only
+enabled on nightly. Many of them are useful for debugging. To get a full listing
+of -Z flags, use -Z help.
+One useful flag is -Z verbose, which generally enables printing more info that
+could be useful for debugging.
+
+When you have an ICE (panic in the compiler), you can set
+RUST_BACKTRACE=1 to get the stack trace of the panic! like in
+normal Rust programs. IIRC backtraces don't work on MinGW,
+sorry. If you have trouble or the backtraces are full of unknown,
+you might want to find some way to use Linux, Mac, or MSVC on Windows.
+In the default configuration, you don't have line numbers enabled, so the
+backtrace looks like this:
+stack backtrace:
+ 0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
+ 1: std::sys_common::backtrace::_print
+ 2: std::panicking::default_hook::{{closure}}
+ 3: std::panicking::default_hook
+ 4: std::panicking::rust_panic_with_hook
+ 5: std::panicking::begin_panic
+ (~~~~ LINES REMOVED BY ME FOR BREVITY ~~~~)
+ 32: rustc_typeck::check_crate
+ 33: <std::thread::local::LocalKey<T>>::with
+ 34: <std::thread::local::LocalKey<T>>::with
+ 35: rustc::ty::context::TyCtxt::create_and_enter
+ 36: rustc_driver::driver::compile_input
+ 37: rustc_driver::run_compiler
+
+If you want line numbers for the stack trace, you can enable debug = true in
+your config.toml and rebuild the compiler (debuginfo-level = 1 will also add
+line numbers, but debug = true gives full debuginfo). Then the backtrace will
+look like this:
+stack backtrace:
+ (~~~~ LINES REMOVED BY ME FOR BREVITY ~~~~)
+ at /home/user/rust/src/librustc_typeck/check/cast.rs:110
+ 7: rustc_typeck::check::cast::CastCheck::check
+ at /home/user/rust/src/librustc_typeck/check/cast.rs:572
+ at /home/user/rust/src/librustc_typeck/check/cast.rs:460
+ at /home/user/rust/src/librustc_typeck/check/cast.rs:370
+ (~~~~ LINES REMOVED BY ME FOR BREVITY ~~~~)
+ 33: rustc_driver::driver::compile_input
+ at /home/user/rust/src/librustc_driver/driver.rs:1010
+ at /home/user/rust/src/librustc_driver/driver.rs:212
+ 34: rustc_driver::run_compiler
+ at /home/user/rust/src/librustc_driver/lib.rs:253
+
+
+If you want to get a backtrace to the point where the compiler emits an
+error message, you can pass the -Z treat-err-as-bug=n, which will make
+the compiler panic on the nth error on delay_span_bug. If you leave
+off =n, the compiler will assume 1 for n and thus panic on the
+first error it encounters.
+This can also help when debugging delay_span_bug calls - it will make
+the first delay_span_bug call panic, which will give you a useful backtrace.
+For example:
+$ cat error.rs
+fn main() {
+ 1 + ();
+}
+
+$ ./build/x86_64-unknown-linux-gnu/stage1/bin/rustc error.rs
+error[E0277]: the trait bound `{integer}: std::ops::Add<()>` is not satisfied
+ --> error.rs:2:7
+ |
+2 | 1 + ();
+ | ^ no implementation for `{integer} + ()`
+ |
+ = help: the trait `std::ops::Add<()>` is not implemented for `{integer}`
+
+error: aborting due to previous error
+
+$ # Now, where does the error above come from?
+$ RUST_BACKTRACE=1 \
+ ./build/x86_64-unknown-linux-gnu/stage1/bin/rustc \
+ error.rs \
+ -Z treat-err-as-bug
+error[E0277]: the trait bound `{integer}: std::ops::Add<()>` is not satisfied
+ --> error.rs:2:7
+ |
+2 | 1 + ();
+ | ^ no implementation for `{integer} + ()`
+ |
+ = help: the trait `std::ops::Add<()>` is not implemented for `{integer}`
+
+error: internal compiler error: unexpected panic
+
+note: the compiler unexpectedly panicked. this is a bug.
+
+note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
+
+note: rustc 1.24.0-dev running on x86_64-unknown-linux-gnu
+
+note: run with `RUST_BACKTRACE=1` for a backtrace
+
+thread 'rustc' panicked at 'encountered error with `-Z treat_err_as_bug',
+/home/user/rust/src/librustc_errors/lib.rs:411:12
+note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose
+backtrace.
+stack backtrace:
+ (~~~ IRRELEVANT PART OF BACKTRACE REMOVED BY ME ~~~)
+ 7: rustc::traits::error_reporting::<impl rustc::infer::InferCtxt<'a, 'tcx>>
+ ::report_selection_error
+ at /home/user/rust/src/librustc_middle/traits/error_reporting.rs:823
+ 8: rustc::traits::error_reporting::<impl rustc::infer::InferCtxt<'a, 'tcx>>
+ ::report_fulfillment_errors
+ at /home/user/rust/src/librustc_middle/traits/error_reporting.rs:160
+ at /home/user/rust/src/librustc_middle/traits/error_reporting.rs:112
+ 9: rustc_typeck::check::FnCtxt::select_obligations_where_possible
+ at /home/user/rust/src/librustc_typeck/check/mod.rs:2192
+ (~~~ IRRELEVANT PART OF BACKTRACE REMOVED BY ME ~~~)
+ 36: rustc_driver::run_compiler
+ at /home/user/rust/src/librustc_driver/lib.rs:253
+$ # Cool, now I have a backtrace for the error
+
+
+These crates are used in compiler for logging:
+
+- log
+- env-logger: check the link to see the full
RUSTC_LOG syntax
+
+The compiler has a lot of debug! calls, which print out logging information
+at many points. These are very useful to at least narrow down the location of
+a bug if not to find it entirely, or just to orient yourself as to why the
+compiler is doing a particular thing.
+To see the logs, you need to set the RUSTC_LOG environment variable to
+your log filter, e.g. to get the logs for a specific module, you can run the
+compiler as RUSTC_LOG=module::path rustc my-file.rs. All debug! output will
+then appear in standard error.
+Note that unless you use a very strict filter, the logger will emit a lot of
+output, so use the most specific module(s) you can (comma-separated if
+multiple). It's typically a good idea to pipe standard error to a file and
+look at the log output with a text editor.
+So to put it together.
+# This puts the output of all debug calls in `librustc_middle/traits` into
+# standard error, which might fill your console backscroll.
+$ RUSTC_LOG=rustc::traits rustc +local my-file.rs
+
+# This puts the output of all debug calls in `librustc_middle/traits` in
+# `traits-log`, so you can then see it with a text editor.
+$ RUSTC_LOG=rustc::traits rustc +local my-file.rs 2>traits-log
+
+# Not recommended. This will show the output of all `debug!` calls
+# in the Rust compiler, and there are a *lot* of them, so it will be
+# hard to find anything.
+$ RUSTC_LOG=debug rustc +local my-file.rs 2>all-log
+
+# This will show the output of all `info!` calls in `rustc_trans`.
+#
+# There's an `info!` statement in `trans_instance` that outputs
+# every function that is translated. This is useful to find out
+# which function triggers an LLVM assertion, and this is an `info!`
+# log rather than a `debug!` log so it will work on the official
+# compilers.
+$ RUSTC_LOG=rustc_trans=info rustc +local my-file.rs
+
+
+While calls to error!, warn! and info! are included in every build of the compiler,
+calls to debug! and trace! are only included in the program if
+debug-assertions=yes is turned on in config.toml (it is
+turned off by default), so if you don't see DEBUG logs, especially
+if you run the compiler with RUSTC_LOG=rustc rustc some.rs and only see
+INFO logs, make sure that debug-assertions=yes is turned on in your
+config.toml.
+In some cases, just setting it will not trigger a rebuild,
+so if you changed it and you already have a compiler built, you might
+want to call x.py clean to force one.
+
+Because calls to debug! are removed by default, in most cases, don't worry
+about adding "unnecessary" calls to debug! and leaving them in code you
+commit - they won't slow down the performance of what we ship, and if they
+helped you pinning down a bug, they will probably help someone else with a
+different one.
+A loosely followed convention is to use debug!("foo(...)") at the start of
+a function foo and debug!("foo: ...") within the function. Another
+loosely followed convention is to use the {:?} format specifier for debug
+logs.
+One thing to be careful of is expensive operations in logs.
+If in the module rustc::foo you have a statement
+debug!("{:?}", random_operation(tcx));
+
+Then if someone runs a debug rustc with RUSTC_LOG=rustc::bar, then
+random_operation() will run.
+This means that you should not put anything too expensive or likely to crash
+there - that would annoy anyone who wants to use logging for their own module.
+No-one will know it until someone tries to use logging to find another bug.
+
+Some compiler options for debugging specific features yield graphviz graphs -
+e.g. the #[rustc_mir(borrowck_graphviz_postflow="suffix.dot")] attribute
+dumps various borrow-checker dataflow graphs.
+These all produce .dot files. To view these files, install graphviz (e.g.
+apt-get install graphviz) and then run the following commands:
+$ dot -T pdf maybe_init_suffix.dot > maybe_init_suffix.pdf
+$ firefox maybe_init_suffix.pdf # Or your favorite pdf viewer
+
+
+The cargo-bisect-rustc tool can be used as a quick and easy way to
+find exactly which PR caused a change in rustc behavior. It automatically
+downloads rustc PR artifacts and tests them against a project you provide
+until it finds the regression. You can then look at the PR to get more context
+on why it was changed. See this tutorial on how to use
+it.
+
+The rustup-toolchain-install-master tool by kennytm can be used to
+download the artifacts produced by Rust's CI for a specific SHA1 -- this
+basically corresponds to the successful landing of some PR -- and then sets
+them up for your local use. This also works for artifacts produced by @bors try. This is helpful when you want to examine the resulting build of a PR
+without doing the build yourself.
+
+The (permanently) unstable #[rustc_layout] attribute can be used to dump
+the Layout of the type it is attached to. For example:
+
+#![allow(unused_variables)]
+#![feature(rustc_attrs)]
+
+fn main() {
+#[rustc_layout(debug)]
+type T<'a> = &'a u32;
+}
+
+Will emit the following:
+error: layout_of(&'a u32) = Layout {
+ fields: Primitive,
+ variants: Single {
+ index: 0,
+ },
+ abi: Scalar(
+ Scalar {
+ value: Pointer,
+ valid_range: 1..=18446744073709551615,
+ },
+ ),
+ largest_niche: Some(
+ Niche {
+ offset: Size {
+ raw: 0,
+ },
+ scalar: Scalar {
+ value: Pointer,
+ valid_range: 1..=18446744073709551615,
+ },
+ },
+ ),
+ align: AbiAndPrefAlign {
+ abi: Align {
+ pow2: 3,
+ },
+ pref: Align {
+ pow2: 3,
+ },
+ },
+ size: Size {
+ raw: 8,
+ },
+}
+ --> src/lib.rs:4:1
+ |
+4 | type T<'a> = &'a u32;
+ | ^^^^^^^^^^^^^^^^^^^^^
+
+error: aborting due to previous error
+
+
+This section talks about how to profile the compiler and find out where it spends its time.
+Depending on what you're trying to measure, there are several different approaches:
+
+-
+
If you want to see if a PR improves or regresses compiler performance:
+
+- The rustc-perf project makes this easy and can be triggered to run on a PR via the
@rustc-perf bot.
+
+
+-
+
If you want a medium-to-high level overview of where rustc is spending its time:
+
+- The
-Zself-profile flag and measureme tools offer a query-based approach to profiling.
+See their docs for more information.
+
+
+-
+
If you want function level performance data or even just more details than the above approaches:
+
+- Consider using a native code profiler such as perf.
+
+
+
+
+This is a guide for how to profile rustc with perf.
+
+
+- Get a clean checkout of rust-lang/master, or whatever it is you want
+to profile.
+- Set the following settings in your
config.toml:
+
+debuginfo-level = 1 - enables line debuginfojemalloc = false - lets you do memory use profiling with valgrind- leave everything else the defaults
+
+
+- Run
./x.py build to get a full build
+- Make a rustup toolchain pointing to that result
+
+
+
+
+perf is an excellent tool on linux that can be used to gather and
+analyze all kinds of information. Mostly it is used to figure out
+where a program spends its time. It can also be used for other sorts
+of events, though, like cache misses and so forth.
+
+The basic perf command is this:
+perf record -F99 --call-graph dwarf XXX
+
+The -F99 tells perf to sample at 99 Hz, which avoids generating too
+much data for longer runs (why 99 Hz you ask? It is often chosen
+because it is unlikely to be in lockstep with other periodic
+activity). The --call-graph dwarf tells perf to get call-graph
+information from debuginfo, which is accurate. The XXX is the
+command you want to profile. So, for example, you might do:
+perf record -F99 --call-graph dwarf cargo +<toolchain> rustc
+
+to run cargo -- here <toolchain> should be the name of the toolchain
+you made in the beginning. But there are some things to be aware of:
+
+- You probably don't want to profile the time spend building
+dependencies. So something like
cargo build; cargo clean -p $C may
+be helpful (where $C is the crate name)
+
+- Though usually I just do
touch src/lib.rs and rebuild instead. =)
+
+
+- You probably don't want incremental messing about with your
+profile. So something like
CARGO_INCREMENTAL=0 can be helpful.
+
+
+Often we want to analyze a specific test from perf.rust-lang.org. To
+do that, the first step is to clone
+the rustc-perf repository:
+git clone https://github.com/rust-lang/rustc-perf
+
+
+Once you've cloned the repo, you can use the collector executable to
+do profiling for you! You can find
+instructions in the rustc-perf readme.
+For example, to measure the clap-rs test, you might do:
+./target/release/collector \
+ --output-repo /path/to/place/output \
+ profile perf-record \
+ --rustc /path/to/rustc/executable/from/your/build/directory \
+ --cargo `which cargo` \
+ --filter clap-rs \
+ --builds Check \
+
+You can also use that same command to use cachegrind or other profiling tools.
+
+If you prefer to run things manually, that is also possible. You first
+need to find the source for the test you want. Sources for the tests
+are found in the collector/benchmarks directory. So let's go
+into the directory of a specific test; we'll use clap-rs as an
+example:
+cd collector/benchmarks/clap-rs
+
+In this case, let's say we want to profile the cargo check
+performance. In that case, I would first run some basic commands to
+build the dependencies:
+# Setup: first clean out any old results and build the dependencies:
+cargo +<toolchain> clean
+CARGO_INCREMENTAL=0 cargo +<toolchain> check
+
+(Again, <toolchain> should be replaced with the name of the
+toolchain we made in the first step.)
+Next: we want record the execution time for just the clap-rs crate,
+running cargo check. I tend to use cargo rustc for this, since it
+also allows me to add explicit flags, which we'll do later on.
+touch src/lib.rs
+CARGO_INCREMENTAL=0 perf record -F99 --call-graph dwarf cargo rustc --profile check --lib
+
+Note that final command: it's a doozy! It uses the cargo rustc
+command, which executes rustc with (potentially) additional options;
+the --profile check and --lib options specify that we are doing a
+cargo check execution, and that this is a library (not a binary).
+At this point, we can use perf tooling to analyze the results. For example:
+perf report
+
+will open up an interactive TUI program. In simple cases, that can be
+helpful. For more detailed examination, the perf-focus tool
+can be helpful; it is covered below.
+A note of caution. Each of the rustc-perf tests is its own special
+snowflake. In particular, some of them are not libraries, in which
+case you would want to do touch src/main.rs and avoid passing
+--lib. I'm not sure how best to tell which test is which to be
+honest.
+
+If you want to profile an NLL run, you can just pass extra options to
+the cargo rustc command, like so:
+touch src/lib.rs
+CARGO_INCREMENTAL=0 perf record -F99 --call-graph dwarf cargo rustc --profile check --lib -- -Zborrowck=mir
+
+
+Once you've gathered a perf profile, we want to get some information
+about it. For this, I personally use perf focus. It's a kind of
+simple but useful tool that lets you answer queries like:
+
+- "how much time was spent in function F" (no matter where it was called from)
+- "how much time was spent in function F when it was called from G"
+- "how much time was spent in function F excluding time spent in G"
+- "what functions does F call and how much time does it spend in them"
+
+To understand how it works, you have to know just a bit about
+perf. Basically, perf works by sampling your process on a regular
+basis (or whenever some event occurs). For each sample, perf gathers a
+backtrace. perf focus lets you write a regular expression that tests
+which functions appear in that backtrace, and then tells you which
+percentage of samples had a backtrace that met the regular
+expression. It's probably easiest to explain by walking through how I
+would analyze NLL performance.
+
+You can install perf-focus using cargo install:
+cargo install perf-focus
+
+
+Let's say we've gathered the NLL data for a test. We'd like to know
+how much time it is spending in the MIR borrow-checker. The "main"
+function of the MIR borrowck is called do_mir_borrowck, so we can do
+this command:
+$ perf focus '{do_mir_borrowck}'
+Matcher : {do_mir_borrowck}
+Matches : 228
+Not Matches: 542
+Percentage : 29%
+
+The '{do_mir_borrowck}' argument is called the matcher. It
+specifies the test to be applied on the backtrace. In this case, the
+{X} indicates that there must be some function on the backtrace
+that meets the regular expression X. In this case, that regex is
+just the name of the function we want (in fact, it's a subset of the name;
+the full name includes a bunch of other stuff, like the module
+path). In this mode, perf-focus just prints out the percentage of
+samples where do_mir_borrowck was on the stack: in this case, 29%.
+A note about c++filt. To get the data from perf, perf focus
+currently executes perf script (perhaps there is a better
+way...). I've sometimes found that perf script outputs C++ mangled
+names. This is annoying. You can tell by running perf script | head yourself — if you see names like 5rustc6middle instead of
+rustc::middle, then you have the same problem. You can solve this
+by doing:
+perf script | c++filt | perf focus --from-stdin ...
+
+This will pipe the output from perf script through c++filt and
+should mostly convert those names into a more friendly format. The
+--from-stdin flag to perf focus tells it to get its data from
+stdin, rather than executing perf focus. We should make this more
+convenient (at worst, maybe add a c++filt option to perf focus, or
+just always use it — it's pretty harmless).
+
+Perhaps we'd like to know how much time MIR borrowck spends in the
+trait checker. We can ask this using a more complex regex:
+$ perf focus '{do_mir_borrowck}..{^rustc::traits}'
+Matcher : {do_mir_borrowck},..{^rustc::traits}
+Matches : 12
+Not Matches: 1311
+Percentage : 0%
+
+Here we used the .. operator to ask "how often do we have
+do_mir_borrowck on the stack and then, later, some function whose
+name begins with rustc::traits?" (basically, code in that module). It
+turns out the answer is "almost never" — only 12 samples fit that
+description (if you ever see no samples, that often indicates your
+query is messed up).
+If you're curious, you can find out exactly which samples by using the
+--print-match option. This will print out the full backtrace for
+each sample. The | at the front of the line indicates the part that
+the regular expression matched.
+
+Often we want to do a more "explorational" queries. Like, we know that
+MIR borrowck is 29% of the time, but where does that time get spent?
+For that, the --tree-callees option is often the best tool. You
+usually also want to give --tree-min-percent or
+--tree-max-depth. The result looks like this:
+$ perf focus '{do_mir_borrowck}' --tree-callees --tree-min-percent 3
+Matcher : {do_mir_borrowck}
+Matches : 577
+Not Matches: 746
+Percentage : 43%
+
+Tree
+| matched `{do_mir_borrowck}` (43% total, 0% self)
+: | rustc_mir::borrow_check::nll::compute_regions (20% total, 0% self)
+: : | rustc_mir::borrow_check::nll::type_check::type_check_internal (13% total, 0% self)
+: : : | core::ops::function::FnOnce::call_once (5% total, 0% self)
+: : : : | rustc_mir::borrow_check::nll::type_check::liveness::generate (5% total, 3% self)
+: : : | <rustc_mir::borrow_check::nll::type_check::TypeVerifier<'a, 'b, 'tcx> as rustc::mir::visit::Visitor<'tcx>>::visit_mir (3% total, 0% self)
+: | rustc::mir::visit::Visitor::visit_mir (8% total, 6% self)
+: | <rustc_mir::borrow_check::MirBorrowckCtxt<'cx, 'tcx> as rustc_mir::dataflow::DataflowResultsConsumer<'cx, 'tcx>>::visit_statement_entry (5% total, 0% self)
+: | rustc_mir::dataflow::do_dataflow (3% total, 0% self)
+
+What happens with --tree-callees is that
+
+- we find each sample matching the regular expression
+- we look at the code that is occurs after the regex match and try
+to build up a call tree
+
+The --tree-min-percent 3 option says "only show me things that take
+more than 3% of the time. Without this, the tree often gets really
+noisy and includes random stuff like the innards of
+malloc. --tree-max-depth can be useful too, it just limits how many
+levels we print.
+For each line, we display the percent of time in that function
+altogether ("total") and the percent of time spent in just that
+function and not some callee of that function (self). Usually
+"total" is the more interesting number, but not always.
+
+By default, all in perf-focus are relative to the total program
+execution. This is useful to help you keep perspective — often as
+we drill down to find hot spots, we can lose sight of the fact that,
+in terms of overall program execution, this "hot spot" is actually not
+important. It also ensures that percentages between different queries
+are easily compared against one another.
+That said, sometimes it's useful to get relative percentages, so perf focus offers a --relative option. In this case, the percentages are
+listed only for samples that match (vs all samples). So for example we
+could get our percentages relative to the borrowck itself
+like so:
+$ perf focus '{do_mir_borrowck}' --tree-callees --relative --tree-max-depth 1 --tree-min-percent 5
+Matcher : {do_mir_borrowck}
+Matches : 577
+Not Matches: 746
+Percentage : 100%
+
+Tree
+| matched `{do_mir_borrowck}` (100% total, 0% self)
+: | rustc_mir::borrow_check::nll::compute_regions (47% total, 0% self) [...]
+: | rustc::mir::visit::Visitor::visit_mir (19% total, 15% self) [...]
+: | <rustc_mir::borrow_check::MirBorrowckCtxt<'cx, 'tcx> as rustc_mir::dataflow::DataflowResultsConsumer<'cx, 'tcx>>::visit_statement_entry (13% total, 0% self) [...]
+: | rustc_mir::dataflow::do_dataflow (8% total, 1% self) [...]
+
+Here you see that compute_regions came up as "47% total" — that
+means that 47% of do_mir_borrowck is spent in that function. Before,
+we saw 20% — that's because do_mir_borrowck itself is only 43% of
+the total time (and .47 * .43 = .20).
+This file offers some tips on the coding conventions for rustc. This
+chapter covers formatting, coding for correctness,
+using crates from crates.io, and some tips on
+structuring your PR for easy review.
+
+
+rustc is slowly moving towards the Rust standard coding style;
+at the moment, however, it follows a rather more chaotic style. We
+do have some mandatory formatting conventions, which are automatically
+enforced by a script we affectionately call the "tidy" script. The
+tidy script runs automatically when you do ./x.py test and can be run
+in isolation with ./x.py test tidy.
+
+
+In the past, files began with a copyright and license notice. Please omit
+this notice for new files licensed under the standard terms (dual
+MIT/Apache-2.0).
+All of the copyright notices should be gone by now, but if you come across one
+in the rust-lang/rust repo, feel free to open a PR to remove it.
+
+Lines should be at most 100 characters. It's even better if you can
+keep things to 80.
+Ignoring the line length limit. Sometimes – in particular for
+tests – it can be necessary to exempt yourself from this limit. In
+that case, you can add a comment towards the top of the file (after
+the copyright notice) like so:
+
+#![allow(unused_variables)]
+fn main() {
+// ignore-tidy-linelength
+}
+
+
+Prefer 4-space indent.
+
+
+Beyond formatting, there are a few other tips that are worth
+following.
+
+Using _ in a match is convenient, but it means that when new
+variants are added to the enum, they may not get handled correctly.
+Ask yourself: if a new variant were added to this enum, what's the
+chance that it would want to use the _ code, versus having some
+other treatment? Unless the answer is "low", then prefer an
+exhaustive match. (The same advice applies to if let and while let, which are effectively tests for a single variant.)
+
+As a useful tool to yourself, you can insert a // TODO comment
+for something that you want to get back to before you land your PR:
+fn do_something() {
+ if something_else {
+ unimplemented!(); // TODO write this
+ }
+}
+
+The tidy script will report an error for a // TODO comment, so this
+code would not be able to land until the TODO is fixed (or removed).
+This can also be useful in a PR as a way to signal from one commit that you are
+leaving a bug that a later commit will fix:
+if foo {
+ return true; // TODO wrong, but will be fixed in a later commit
+}
+
+
+
+It is allowed to use crates from crates.io, though external
+dependencies should not be added gratuitously. All such crates must
+have a suitably permissive license. There is an automatic check which
+inspects the Cargo metadata to ensure this.
+
+
+How you prepare the commits in your PR can make a big difference for the
+reviewer. Here are some tips.
+Isolate "pure refactorings" into their own commit. For example, if
+you rename a method, then put that rename into its own commit, along
+with the renames of all the uses.
+More commits is usually better. If you are doing a large change,
+it's almost always better to break it up into smaller steps that can
+be independently understood. The one thing to be aware of is that if
+you introduce some code following one strategy, then change it
+dramatically (versus adding to it) in a later commit, that
+'back-and-forth' can be confusing.
+Only run rustfmt on new content. One day, we might enforce formatting
+for the rust-lang/rust repo. Meanwhile, we prefer that rustfmt not be run
+on existing code as that will generate large diffs and will make git blame
+harder to sift through. However, running rustfmt on new content, e.g. a
+new file or a largely new part of a file is ok. Small formatting adjustments
+nearby code you are already changing for other purposes are also ok.
+No merges. We do not allow merge commits into our history, other
+than those by bors. If you get a merge conflict, rebase instead via a
+command like git rebase -i rust-lang/master (presuming you use the
+name rust-lang for your remote).
+Individual commits do not have to build (but it's nice). We do not
+require that every intermediate commit successfully builds – we only
+expect to be able to bisect at a PR level. However, if you can make
+individual commits build, that is always helpful.
+
+Apart from normal Rust style/naming conventions, there are also some specific
+to the compiler.
+
+-
+
cx tends to be short for "context" and is often used as a suffix. For
+example, tcx is a common name for the Typing Context.
+
+-
+
'tcx is used as the lifetime name for the Typing Context.
+
+-
+
Because crate is a keyword, if you need a variable to represent something
+crate-related, often the spelling is changed to krate.
+
+
+
+The rust compiler supports building with some dependencies from crates.io.
+For example, log and env_logger come from crates.io.
+In general, you should avoid adding dependencies to the compiler for several
+reasons:
+
+- The dependency may not be high quality or well-maintained, whereas we want
+the compiler to be high-quality.
+- The dependency may not be using a compatible license.
+- The dependency may have transitive dependencies that have one of the above
+problems.
+
+TODO: what is the vetting process?
+
+The tidy tool has a whitelist of crates that are allowed. To add a
+dependency that is not already in the compiler, you will need to add it to this
+whitelist.
+
+A lot of effort has been put into making rustc have great error messages.
+This chapter is about how to emit compile errors and lints from the compiler.
+
+The main parts of a diagnostic error are the following:
+error[E0000]: main error message
+ --> file.rs:LL:CC
+ |
+LL | <code>
+ | -^^^^- secondary label
+ | |
+ | primary label
+ |
+ = note: note without a `Span`, created with `.note`
+note: sub-diagnostic message for `.span_note`
+ --> file.rs:LL:CC
+ |
+LL | more code
+ | ^^^^
+
+
+- Description (
error, warning, etc.).
+- Code (for example, for "mismatched types", it is
E0308). It helps
+users get more information about the current error through an extended
+description of the problem in the error code index.
+- Message. It is the main description of the problem. It should be general and
+able to stand on its own, so that it can make sense even in isolation.
+- Diagnostic window. This contains several things:
+
+- The path, line number and column of the beginning of the primary span.
+- The users' affected code and its surroundings.
+- Primary and secondary spans underlying the users' code. These spans can
+optionally contain one or more labels.
+
+- Primary spans should have enough text to describe the problem in such a
+way that if it where the only thing being displayed (for example, in an
+IDE) it would still make sense. Because it is "spatially aware" (it
+points at the code), it can generally be more succinct than the error
+message.
+- If cluttered output can be foreseen in cases when multiple span labels
+overlap, it is a good idea to tweak the output appropriately. For
+example, the
if/else arms have incompatible types error uses different
+spans depending on whether the arms are all in the same line, if one of
+the arms is empty and if none of those cases applies.
+
+
+
+
+- Sub-diagnostics. Any error can have multiple sub-diagnostics that look
+similar to the main part of the error. These are used for cases where the
+order of the explanation might not correspond with the order of the code. If
+the order of the explanation can be "order free", leveraging secondary labels
+in the main diagnostic is preferred, as it is typically less verbose.
+
+The text should be matter of fact and avoid capitalization and periods, unless
+multiple sentences are needed:
+error: the fobrulator needs to be krontrificated
+
+When code or an identifier must appear in an message or label, it should be
+surrounded with single acute accents `.
+
+Some errors include long form descriptions. They may be viewed with the
+--explain flag, or via the error index. Each explanation comes with an
+example of how to trigger it and advice on how to fix it.
+Please read RFC 1567 for details on how to format and write long error
+codes.
+The descriptions are written in markdown, and all of them are linked in the
+librustc_error_codes crate.
+TODO: When should an error use an error code, and when shouldn't it?
+
+Some messages are emitted via lints, where the user can control the
+level. Most diagnostics are hard-coded such that the user cannot control the
+level.
+Usually it is obvious whether a diagnostic should be "fixed" or a lint, but
+there are some grey areas.
+Here are a few examples:
+
+- Borrow checker errors: these are fixed errors. The user cannot adjust the
+level of these diagnostics to silence the borrow checker.
+- Dead code: this is a lint. While the user probably doesn't want dead code in
+their crate, making this a hard error would make refactoring and development
+very painful.
+- safe_packed_borrows future compatibility warning:
+this is a silencable lint related to safety. It was judged that the making
+this a hard (fixed) error would cause too much breakage, so instead a
+warning is emitted that eventually will be turned into a hard error.
+
+Hard-coded warnings (those using the span_warn methods) should be avoided
+for normal code, preferring to use lints instead. Some cases, such as warnings
+with CLI flags, will require the use of hard-coded warnings.
+See the deny lint level below for guidelines when to
+use an error-level lint instead of a fixed error.
+
+
+- Write in plain simple English. If your message, when shown on a – possibly
+small – screen (which hasn't been cleaned for a while), cannot be understood
+by a normal programmer, who just came out of bed after a night partying,
+it's too complex.
+Error, Warning, Note, and Help messages start with a lowercase
+letter and do not end with punctuation.- Error messages should be succinct. Users will see these error messages many
+times, and more verbose descriptions can be viewed with the
--explain
+flag. That said, don't make it so terse that it's hard to understand.
+- The word "illegal" is illegal. Prefer "invalid" or a more specific word
+instead.
+- Errors should document the span of code where they occur – the
+
rustc_errors::diagnostic_builder::DiagnosticBuilder span_*
+methods allow to easily do this. Also note other spans that have
+contributed to the error if the span isn't too large.
+- When emitting a message with span, try to reduce the span to the smallest
+amount possible that still signifies the issue
+- Try not to emit multiple error messages for the same error. This may require
+detecting duplicates.
+- When the compiler has too little information for a specific error message,
+consult with the compiler team to add new attributes for library code that
+allow adding more information. For example see
+
#[rustc_on_unimplemented]. Use these
+annotations when available!
+- Keep in mind that Rust's learning curve is rather steep, and that the
+compiler messages are an important learning tool.
+- When talking about the compiler, call it
the compiler, not Rust or
+rustc.
+
+
+From RFC 0344, lint names should be consistent, with the following
+guidelines:
+The basic rule is: the lint name should make sense when read as "allow
+lint-name" or "allow lint-name items". For example, "allow
+deprecated items" and "allow dead_code" makes sense, while "allow
+unsafe_block" is ungrammatical (should be plural).
+
+-
+
Lint names should state the bad thing being checked for, e.g. deprecated,
+so that #[allow(deprecated)] (items) reads correctly. Thus ctypes is not
+an appropriate name; improper_ctypes is.
+
+-
+
Lints that apply to arbitrary items (like the stability lints) should just
+mention what they check for: use deprecated rather than
+deprecated_items. This keeps lint names short. (Again, think "allow
+lint-name items".)
+
+-
+
If a lint applies to a specific grammatical class, mention that class and
+use the plural form: use unused_variables rather than unused_variable.
+This makes #[allow(unused_variables)] read correctly.
+
+-
+
Lints that catch unnecessary, unused, or useless aspects of code should use
+the term unused, e.g. unused_imports, unused_typecasts.
+
+-
+
Use snake case in the same way you would for function names.
+
+
+
+Guidelines for different diagnostic levels:
+
+-
+
error: emitted when the compiler detects a problem that makes it unable to
+compile the program, either because the program is invalid or the programmer
+has decided to make a specific warning into an error.
+
+-
+
warning: emitted when the compiler detects something odd about a program.
+Care should be taken when adding warnings to avoid warning fatigue, and
+avoid false-positives where there really isn't a problem with the code. Some
+examples of when it is appropriate to issue a warning:
+
+- A situation where the user should take action, such as swap out a
+deprecated item, or use a
Result, but otherwise doesn't prevent
+compilation.
+- Unnecessary syntax that can be removed without affecting the semantics of
+the code. For example, unused code, or unnecessary
unsafe.
+- Code that is very likely to be incorrect, dangerous, or confusing, but the
+language technically allows, and is not ready or confident enough to make
+an error. For example
unused_comparisons (out of bounds comparisons) or
+bindings_with_variant_name (the user likely did not intend to create a
+binding in a pattern).
+- Future-incompatible lints, where something was
+accidentally or erroneously accepted in the past, but rejecting would
+cause excessive breakage in the ecosystem.
+- Stylistic choices. For example, camel or snake case, or the
dyn trait
+warning in the 2018 edition. These have a high bar to be added, and should
+only be used in exceptional circumstances. Other stylistic choices should
+either be allow-by-default lints, or part of other tools like Clippy or
+rustfmt.
+
+
+-
+
help: emitted following an error or warning to give additional
+information to the user about how to solve their problem. These messages
+often include a suggestion string and rustc_errors::Applicability
+confidence level to guide automated source fixes by tools. See the
+Suggestions section for more details.
+The error or warning portion should not suggest how to fix the problem,
+only the "help" sub-diagnostic should.
+
+-
+
note: emitted to identify additional circumstances and parts of the code
+that caused the warning or error. For example, the borrow checker will note
+any previous conflicting borrows.
+
+
+Not to be confused with lint levels, whose guidelines are:
+
+-
+
forbid: Lints should never default to forbid.
+
+-
+
deny: Equivalent to error diagnostic level. Some examples:
+
+- A future-incompatible or edition-based lint that has graduated from the
+warning level.
+- Something that has an extremely high confidence that is incorrect, but
+still want an escape hatch to allow it to pass.
+
+
+-
+
warn: Equivalent to the warning diagnostic level. See warning above
+for guidelines.
+
+-
+
allow: Examples of the kinds of lints that should default to allow:
+
+- The lint has a too high false positive rate.
+- The lint is too opinionated.
+- The lint is experimental.
+- The lint is used for enforcing something that is not normally enforced.
+For example, the
unsafe_code lint can be used to prevent usage of unsafe
+code.
+
+
+
+More information about lint levels can be found in the rustc
+book and the reference.
+
+
+There are two main ways to find where a given error is emitted:
+
+grep for either a sub-part of the error message/label or error code. This
+usually works well and is straightforward, but there are some cases where
+the error emitting code is removed from the code where the error is
+constructed behind a relatively deep call-stack. Even then, it is a good way
+to get your bearings.- Invoking
rustc with the nightly-only flag -Ztreat-err-as-bug=1, which
+will treat the first error being emitted as an Internal Compiler Error, which
+allows you to use the environment variable RUST_BACKTRACE=full to get a
+stack trace at the point the error has been emitted. Change the 1 to
+something else if you whish to trigger on a later error. Some limitations
+with this approach is that some calls get elided from the stack trace because
+they get inlined in the compiled rustc, and the same problem we faced with
+the prior approach, where the construction of the error is far away from
+where it is emitted. In some cases we buffer multiple errors in order to
+emit them in order.
+
+The regular development practices apply: judicious use of debug!() statements
+and use of a debugger to trigger break points in order to figure out in what
+order things are happening.
+
+Span is the primary data structure in rustc used to represent a
+location in the code being compiled. Spans are attached to most constructs in
+HIR and MIR, allowing for more informative error reporting.
+A Span can be looked up in a SourceMap to get a "snippet"
+useful for displaying errors with span_to_snippet and other
+similar methods on the SourceMap.
+
+The rustc_errors crate defines most of the utilities used for
+reporting errors.
+Session and ParseSess have
+methods (or fields with methods) that allow reporting errors. These methods
+usually have names like span_err or struct_span_err or span_warn, etc...
+There are lots of them; they emit different types of "errors", such as
+warnings, errors, fatal errors, suggestions, etc.
+In general, there are two classes of such methods: ones that emit an error
+directly and ones that allow finer control over what to emit. For example,
+span_err emits the given error message at the given Span, but
+struct_span_err instead returns a
+DiagnosticBuilder.
+DiagnosticBuilder allows you to add related notes and suggestions to an error
+before emitting it by calling the emit method. (Failing to either
+emit or cancel a DiagnosticBuilder will result in an ICE.) See the
+docs for more info on what you can do.
+// Get a DiagnosticBuilder. This does _not_ emit an error yet.
+let mut err = sess.struct_span_err(sp, "oh no! this is an error!");
+
+// In some cases, you might need to check if `sp` is generated by a macro to
+// avoid printing weird errors about macro-generated code.
+
+if let Ok(snippet) = sess.source_map().span_to_snippet(sp) {
+ // Use the snippet to generate a suggested fix
+ err.span_suggestion(suggestion_sp, "try using a qux here", format!("qux {}", snippet));
+} else {
+ // If we weren't able to generate a snippet, then emit a "help" message
+ // instead of a concrete "suggestion". In practice this is unlikely to be
+ // reached.
+ err.span_help(suggestion_sp, "you could use a qux here instead");
+}
+
+// emit the error
+err.emit();
+
+
+In addition to telling the user exactly why their code is wrong, it's
+oftentimes furthermore possible to tell them how to fix it. To this end,
+DiagnosticBuilder offers a structured suggestions API, which formats code
+suggestions pleasingly in the terminal, or (when the --error-format json flag
+is passed) as JSON for consumption by tools, most notably the Rust Language
+Server and rustfix.
+Not all suggestions should be applied mechanically, they have a degree of
+confidence in the suggested code, from high
+(Applicability::MachineApplicable) to low (Applicability::MaybeIncorrect).
+Be conservative when choosing the level. Use the
+span_suggestion method of DiagnosticBuilder to
+make a suggestion. The last argument provides a hint to tools whether
+the suggestion is mechanically applicable or not.
+Suggestions point to one or more spans with corresponding code that will
+replace their current content.
+The message that accompanies them should be understandable in the following
+contexts:
+
+- shown as an independent sub-diagnostic (this is the default output)
+- shown as a label pointing at the affected span (this is done automatically if
+some heuristics for verbosity are met)
+- shown as a
help sub-diagnostic with no content (used for cases where the
+suggestion is obvious from the text, but we still want to let tools to apply
+them))
+- not shown (used for very obvious cases, but we still want to allow tools to
+apply them)
+
+For example, to make our qux suggestion machine-applicable, we would do:
+let mut err = sess.struct_span_err(sp, "oh no! this is an error!");
+
+if let Ok(snippet) = sess.source_map().span_to_snippet(sp) {
+ err.span_suggestion(
+ suggestion_sp,
+ "try using a qux here",
+ format!("qux {}", snippet),
+ Applicability::MachineApplicable,
+ );
+} else {
+ err.span_help(suggestion_sp, "you could use a qux here instead");
+}
+
+err.emit();
+
+This might emit an error like
+$ rustc mycode.rs
+error[E0999]: oh no! this is an error!
+ --> mycode.rs:3:5
+ |
+3 | sad()
+ | ^ help: try using a qux here: `qux sad()`
+
+error: aborting due to previous error
+
+For more information about this error, try `rustc --explain E0999`.
+
+In some cases, like when the suggestion spans multiple lines or when there are
+multiple suggestions, the suggestions are displayed on their own:
+error[E0999]: oh no! this is an error!
+ --> mycode.rs:3:5
+ |
+3 | sad()
+ | ^
+help: try using a qux here:
+ |
+3 | qux sad()
+ | ^^^
+
+error: aborting due to previous error
+
+For more information about this error, try `rustc --explain E0999`.
+
+The possible values of Applicability are:
+
+MachineApplicable: Can be applied mechanically.HasPlaceholders: Cannot be applied mechanically because it has placeholder
+text in the suggestions. For example, "Try adding a type: `let x:
+<type>`".MaybeIncorrect: Cannot be applied mechanically because the suggestion may
+or may not be a good one.Unspecified: Cannot be applied mechanically because we don't know which
+of the above cases it falls into.
+
+The compiler linting infrastructure is defined in the rustc::lint
+module.
+
+The built-in compiler lints are defined in the rustc_lint
+crate.
+Every lint is implemented via a struct that implements the LintPass trait
+(you also implement one of the more specific lint pass traits, either
+EarlyLintPass or LateLintPass). The trait implementation allows you to
+check certain syntactic constructs as the linter walks the source code. You can
+then choose to emit lints in a very similar way to compile errors.
+You also declare the metadata of a particular lint via the declare_lint!
+macro. This includes the name, the default level, a short description, and some
+more details.
+Note that the lint and the lint pass must be registered with the compiler.
+For example, the following lint checks for uses
+of while true { ... } and suggests using loop { ... } instead.
+// Declare a lint called `WHILE_TRUE`
+declare_lint! {
+ WHILE_TRUE,
+
+ // warn-by-default
+ Warn,
+
+ // This string is the lint description
+ "suggest using `loop { }` instead of `while true { }`"
+}
+
+// This declares a struct and a lint pass, providing a list of associated lints. The
+// compiler currently doesn't use the associated lints directly (e.g., to not
+// run the pass or otherwise check that the pass emits the appropriate set of
+// lints). However, it's good to be accurate here as it's possible that we're
+// going to register the lints via the get_lints method on our lint pass (that
+// this macro generates).
+declare_lint_pass!(WhileTrue => [WHILE_TRUE]);
+
+// Helper function for `WhileTrue` lint.
+// Traverse through any amount of parenthesis and return the first non-parens expression.
+fn pierce_parens(mut expr: &ast::Expr) -> &ast::Expr {
+ while let ast::ExprKind::Paren(sub) = &expr.kind {
+ expr = sub;
+ }
+ expr
+}
+
+// `EarlyLintPass` has lots of methods. We only override the definition of
+// `check_expr` for this lint because that's all we need, but you could
+// override other methods for your own lint. See the rustc docs for a full
+// list of methods.
+impl EarlyLintPass for WhileTrue {
+ fn check_expr(&mut self, cx: &EarlyContext<'_>, e: &ast::Expr) {
+ if let ast::ExprKind::While(cond, ..) = &e.kind {
+ if let ast::ExprKind::Lit(ref lit) = pierce_parens(cond).kind {
+ if let ast::LitKind::Bool(true) = lit.kind {
+ if !lit.span.from_expansion() {
+ let msg = "denote infinite loops with `loop { ... }`";
+ let condition_span = cx.sess.source_map().guess_head_span(e.span);
+ cx.struct_span_lint(WHILE_TRUE, condition_span, |lint| {
+ lint.build(msg)
+ .span_suggestion_short(
+ condition_span,
+ "use `loop`",
+ "loop".to_owned(),
+ Applicability::MachineApplicable,
+ )
+ .emit();
+ })
+ }
+ }
+ }
+ }
+ }
+}
+
+
+Sometimes we want to change the behavior of a lint in a new edition. To do this,
+we just add the transition to our invocation of declare_lint!:
+declare_lint! {
+ pub ANONYMOUS_PARAMETERS,
+ Allow,
+ "detects anonymous parameters",
+ Edition::Edition2018 => Warn,
+}
+
+This makes the ANONYMOUS_PARAMETERS lint allow-by-default in the 2015 edition
+but warn-by-default in the 2018 edition.
+A future-incompatible lint should be declared with the @future_incompatible
+additional "field":
+declare_lint! {
+ pub ANONYMOUS_PARAMETERS,
+ Allow,
+ "detects anonymous parameters",
+ @future_incompatible = FutureIncompatibleInfo {
+ reference: "issue #41686 <https://github.com/rust-lang/rust/issues/41686>",
+ edition: Some(Edition::Edition2018),
+ };
+}
+
+If you need a combination of options that's not supported by the
+declare_lint! macro, you can always define your own static with a type of
+&Lint but this is currently linted against in the compiler tree.
+
+
+
+- Create a lint defaulting to warn as normal, with ideally the same error
+message you would normally give.
+- Add a suitable reference, typically an RFC or tracking issue. Go ahead
+and include the full URL, sort items in ascending order of issue numbers.
+- Later, change lint to error.
+- Eventually, remove lint.
+
+
+Lints can be turned on in groups. These groups are declared in the
+register_builtins function in rustc_lint::lib. The
+add_lint_group! macro is used to declare a new group.
+For example,
+add_lint_group!(sess,
+ "nonstandard_style",
+ NON_CAMEL_CASE_TYPES,
+ NON_SNAKE_CASE,
+ NON_UPPER_CASE_GLOBALS);
+
+This defines the nonstandard_style group which turns on the listed lints. A
+user can turn on these lints with a !#[warn(nonstandard_style)] attribute in
+the source code, or by passing -W nonstandard-style on the command line.
+
+On occasion, you may need to define a lint that runs before the linting system
+has been initialized (e.g. during parsing or macro expansion). This is
+problematic because we need to have computed lint levels to know whether we
+should emit a warning or an error or nothing at all.
+To solve this problem, we buffer the lints until the linting system is
+processed. Session and ParseSess both have
+buffer_lint methods that allow you to buffer a lint for later. The linting
+system automatically takes care of handling buffered lints later.
+Thus, to define a lint that runs early in the compilation, one defines a lint
+like normal but invokes the lint with buffer_lint.
+
+The parser (librustc_ast) is interesting in that it cannot have dependencies on
+any of the other librustc* crates. In particular, it cannot depend on
+librustc_middle::lint or librustc_lint, where all of the compiler linting
+infrastructure is defined. That's troublesome!
+To solve this, librustc_ast defines its own buffered lint type, which
+ParseSess::buffer_lint uses. After macro expansion, these buffered lints are
+then dumped into the Session::buffered_lints used by the rest of the compiler.
+
+The compiler accepts an --error-format json flag to output
+diagnostics as JSON objects (for the benefit of tools such as cargo fix or the RLS). It looks like this—
+$ rustc json_error_demo.rs --error-format json
+{"message":"cannot add `&str` to `{integer}`","code":{"code":"E0277","explanation":"\nYou tried to use a type which doesn't implement some trait in a place which\nexpected that trait. Erroneous code example:\n\n```compile_fail,E0277\n// here we declare the Foo trait with a bar method\ntrait Foo {\n fn bar(&self);\n}\n\n// we now declare a function which takes an object implementing the Foo trait\nfn some_func<T: Foo>(foo: T) {\n foo.bar();\n}\n\nfn main() {\n // we now call the method with the i32 type, which doesn't implement\n // the Foo trait\n some_func(5i32); // error: the trait bound `i32 : Foo` is not satisfied\n}\n```\n\nIn order to fix this error, verify that the type you're using does implement\nthe trait. Example:\n\n```\ntrait Foo {\n fn bar(&self);\n}\n\nfn some_func<T: Foo>(foo: T) {\n foo.bar(); // we can now use this method since i32 implements the\n // Foo trait\n}\n\n// we implement the trait on the i32 type\nimpl Foo for i32 {\n fn bar(&self) {}\n}\n\nfn main() {\n some_func(5i32); // ok!\n}\n```\n\nOr in a generic context, an erroneous code example would look like:\n\n```compile_fail,E0277\nfn some_func<T>(foo: T) {\n println!(\"{:?}\", foo); // error: the trait `core::fmt::Debug` is not\n // implemented for the type `T`\n}\n\nfn main() {\n // We now call the method with the i32 type,\n // which *does* implement the Debug trait.\n some_func(5i32);\n}\n```\n\nNote that the error here is in the definition of the generic function: Although\nwe only call it with a parameter that does implement `Debug`, the compiler\nstill rejects the function: It must work with all possible input types. In\norder to make this example compile, we need to restrict the generic type we're\naccepting:\n\n```\nuse std::fmt;\n\n// Restrict the input type to types that implement Debug.\nfn some_func<T: fmt::Debug>(foo: T) {\n println!(\"{:?}\", foo);\n}\n\nfn main() {\n // Calling the method is still fine, as i32 implements Debug.\n some_func(5i32);\n\n // This would fail to compile now:\n // struct WithoutDebug;\n // some_func(WithoutDebug);\n}\n```\n\nRust only looks at the signature of the called function, as such it must\nalready specify all requirements that will be used for every type parameter.\n"},"level":"error","spans":[{"file_name":"json_error_demo.rs","byte_start":50,"byte_end":51,"line_start":4,"line_end":4,"column_start":7,"column_end":8,"is_primary":true,"text":[{"text":" a + b","highlight_start":7,"highlight_end":8}],"label":"no implementation for `{integer} + &str`","suggested_replacement":null,"suggestion_applicability":null,"expansion":null}],"children":[{"message":"the trait `std::ops::Add<&str>` is not implemented for `{integer}`","code":null,"level":"help","spans":[],"children":[],"rendered":null}],"rendered":"error[E0277]: cannot add `&str` to `{integer}`\n --> json_error_demo.rs:4:7\n |\n4 | a + b\n | ^ no implementation for `{integer} + &str`\n |\n = help: the trait `std::ops::Add<&str>` is not implemented for `{integer}`\n\n"}
+{"message":"aborting due to previous error","code":null,"level":"error","spans":[],"children":[],"rendered":"error: aborting due to previous error\n\n"}
+{"message":"For more information about this error, try `rustc --explain E0277`.","code":null,"level":"","spans":[],"children":[],"rendered":"For more information about this error, try `rustc --explain E0277`.\n"}
+
+Note that the output is a series of lines, each of which is a JSON
+object, but the series of lines taken together is, unfortunately, not
+valid JSON, thwarting tools and tricks (such as piping to python3 -m json.tool)
+that require such. (One speculates that this was intentional for LSP
+performance purposes, so that each line/object can be sent to RLS as
+it is flushed?)
+Also note the "rendered" field, which contains the "human" output as a
+string; this was introduced so that UI tests could both make use of
+the structured JSON and see the "human" output (well, sans colors)
+without having to compile everything twice.
+The "human" readable and the json format emitter can be found under
+librustc_errors, both were moved from the librustc_ast crate to the
+librustc_errors crate.
+The JSON emitter defines its own Diagnostic
+struct
+(and sub-structs) for the JSON serialization. Don't confuse this with
+errors::Diagnostic!
+
+The #[rustc_on_unimplemented] attribute allows trait definitions to add specialized
+notes to error messages when an implementation was expected but not found.
+You can refer to the trait's generic arguments by name and to the resolved type using Self.
+For example:
+#![feature(rustc_attrs)]
+
+#[rustc_on_unimplemented="an iterator over elements of type `{A}` \
+ cannot be built from a collection of type `{Self}`"]
+trait MyIterator<A> {
+ fn next(&mut self) -> A;
+}
+
+fn iterate_chars<I: MyIterator<char>>(i: I) {
+ // ...
+}
+
+fn main() {
+ iterate_chars(&[1, 2, 3][..]);
+}
+
+When the user compiles this, they will see the following;
+error[E0277]: the trait bound `&[{integer}]: MyIterator<char>` is not satisfied
+ --> <anon>:14:5
+ |
+14 | iterate_chars(&[1, 2, 3][..]);
+ | ^^^^^^^^^^^^^ an iterator over elements of type `char` cannot be built from a collection of type `&[{integer}]`
+ |
+ = help: the trait `MyIterator<char>` is not implemented for `&[{integer}]`
+ = note: required by `iterate_chars`
+
+rustc_on_unimplemented also supports advanced filtering for better targeting
+of messages, as well as modifying specific parts of the error message. You
+target the text of:
+
+- the main error message (
message)
+- the label (
label)
+- an extra note (
note)
+
+For example, the following attribute
+#[rustc_on_unimplemented(
+ message="message",
+ label="label",
+ note="note"
+)]
+trait MyIterator<A> {
+ fn next(&mut self) -> A;
+}
+
+Would generate the following output:
+error[E0277]: message
+ --> <anon>:14:5
+ |
+14 | iterate_chars(&[1, 2, 3][..]);
+ | ^^^^^^^^^^^^^ label
+ |
+ = note: note
+ = help: the trait `MyIterator<char>` is not implemented for `&[{integer}]`
+ = note: required by `iterate_chars`
+
+To allow more targeted error messages, it is possible to filter the
+application of these fields based on a variety of attributes when using
+on:
+
+crate_local: whether the code causing the trait bound to not be
+fulfilled is part of the user's crate. This is used to avoid suggesting
+code changes that would require modifying a dependency.- Any of the generic arguments that can be substituted in the text can be
+referred by name as well for filtering, like
Rhs="i32", except for
+Self.
+_Self: to filter only on a particular calculated trait resolution, like
+Self="std::iter::Iterator<char>". This is needed because Self is a
+keyword which cannot appear in attributes.direct: user-specified rather than derived obligation.from_method: usable both as boolean (whether the flag is present, like
+crate_local) or matching against a particular method. Currently used
+for try.from_desugaring: usable both as boolean (whether the flag is present)
+or matching against a particular desugaring. The desugaring is identified
+with its variant name in the DesugaringKind enum.
+For example, the Iterator trait can be annotated in the following way:
+#[rustc_on_unimplemented(
+ on(
+ _Self="&str",
+ note="call `.chars()` or `.as_bytes()` on `{Self}"
+ ),
+ message="`{Self}` is not an iterator",
+ label="`{Self}` is not an iterator",
+ note="maybe try calling `.iter()` or a similar method"
+)]
+pub trait Iterator {}
+
+Which would produce the following outputs:
+error[E0277]: `Foo` is not an iterator
+ --> src/main.rs:4:16
+ |
+4 | for foo in Foo {}
+ | ^^^ `Foo` is not an iterator
+ |
+ = note: maybe try calling `.iter()` or a similar method
+ = help: the trait `std::iter::Iterator` is not implemented for `Foo`
+ = note: required by `std::iter::IntoIterator::into_iter`
+
+error[E0277]: `&str` is not an iterator
+ --> src/main.rs:5:16
+ |
+5 | for foo in "" {}
+ | ^^ `&str` is not an iterator
+ |
+ = note: call `.chars()` or `.bytes() on `&str`
+ = help: the trait `std::iter::Iterator` is not implemented for `&str`
+ = note: required by `std::iter::IntoIterator::into_iter`
+
+If you need to filter on multiple attributes, you can use all, any or
+not in the following way:
+#[rustc_on_unimplemented(
+ on(
+ all(_Self="&str", T="std::string::String"),
+ note="you can coerce a `{T}` into a `{Self}` by writing `&*variable`"
+ )
+)]
+pub trait From<T>: Sized { /* ... */ }
+
+
+This page documents some of the machinery around lint registration and how we
+run lints in the compiler.
+The LintStore is the central piece of infrastructure, around which everything
+rotates. It's not available during the early parts of compilation (i.e., before
+TyCtxt) in most code, as we need to fill it in with all of the lints, which can only happen after
+plugin registration.
+
+There are two parts to the linting mechanism within the compiler: lints and lint passes.
+Unfortunately, a lot of the documentation we have refers to both of these as just "lints."
+First, we have the lint declarations themselves: this is where the name and default lint level and
+other metadata come from. These are normally defined by way of the declare_lint! macro, which
+boils down to a static with type &rustc::lint::Lint. We lint against direct declarations without
+the use of the macro today (though this may change in the future, as the macro is somewhat unwieldy
+to add new fields to, like all macros by example).
+Lint declarations don't carry any "state" - they are merely global identifers and descriptions of
+lints. We assert at runtime that they are not registered twice (by lint name).
+Lint passes are the meat of any lint. Notably, there is not a one-to-one relationship between
+lints and lint passes; a lint might not have any lint pass that emits it, it could have many, or
+just one -- the compiler doesn't track whether a pass is in any way associated with a particular
+lint, and frequently lints are emitted as part of other work (e.g., type checking, etc.).
+
+
+The lint store is created and all lints are registered during plugin registration, in
+rustc_interface::register_plugins. There are three 'sources' of lint: the internal lints, plugin
+lints, and rustc_interface::Config register_lints. All are registered here, in
+register_plugins.
+Once the registration is complete, we "freeze" the lint store by placing it in an Lrc. Later in
+the driver, it's passed into the GlobalCtxt constructor where it lives in an immutable form from
+then on.
+Lints are registered via the LintStore::register_lint function. This should
+happen just once for any lint, or an ICE will occur.
+Lint passes are registered separately into one of the categories (pre-expansion,
+early, late, late module). Passes are registered as a closure -- i.e., impl Fn() -> Box<dyn X>, where dyn X is either an early or late lint pass trait
+object. When we run the lint passes, we run the closure and then invoke the lint
+pass methods, which take &mut self -- lint passes can keep track of state
+internally.
+
+Note, these include both rustc-internal lints, and the traditional lints, like, for example the dead
+code lint.
+These are primarily described in two places: rustc::lint::builtin and rustc_lint::builtin. The
+first provides the definitions for the lints themselves, and the latter provides the lint pass
+definitions (and implementations).
+The internal lint registration happens in the rustc_lint::register_builtins function, along with
+the rustc_lint::register_internals function. More generally, the LintStore "constructor"
+function which is the way to get a LintStore in the compiler (you should not construct it
+directly) is rustc_lint::new_lint_store; it calls the registration functions.
+
+This is one of the primary use cases remaining for plugins/drivers. Plugins are given access to the
+mutable LintStore during registration to call any functions they need on the LintStore, just
+like rustc code. Plugins are intended to declare lints with the plugin field set to true (e.g., by
+way of the declare_tool_lint! macro), but this is purely for diagnostics and help text;
+otherwise plugin lints are mostly just as first class as rustc builtin lints.
+
+These are the lints provided by drivers via the rustc_interface::Config register_lints field,
+which is a callback. Drivers should, if finding it already set, call the function currently set
+within the callback they add. The best way for drivers to get access to this is by overriding the
+Callbacks::config function which gives them direct access to the Config structure.
+
+Within the compiler, for performance reasons, we usually do not register dozens
+of lint passes. Instead, we have a single lint pass of each variety
+(e.g. BuiltinCombinedModuleLateLintPass) which will internally call all of the
+individual lint passes; this is because then we get the benefits of static over
+dynamic dispatch for each of the (often empty) trait methods.
+Ideally, we'd not have to do this, since it certainly adds to the complexity of
+understanding the code. However, with the current type-erased lint store
+approach, it is beneficial to do so for performance reasons.
+New lints being added likely want to join one of the existing declarations like
+late_lint_mod_passes in librustc_lint/lib.rs, which would then
+auto-propagate into the other.
+
+We generally try assign each error message a unique code like E0123. These
+codes are defined in the compiler in the diagnostics.rs files found in each
+crate, which basically consist of macros. The codes come in two varieties: those
+that have an extended write-up, and those that do not. Whenever possible, if you
+are making a new code, you should write an extended write-up.
+
+If you want to create a new error, you first need to find the next available
+code. This is a bit tricky since the codes are defined in various crates. To do
+it, run this obscure command:
+./x.py test --stage 0 tidy
+
+This will invoke the tidy script, which generally checks that your code obeys
+our coding conventions. One of those jobs is to check that diagnostic codes are
+indeed unique. Once it is finished with that, tidy will print out the lowest
+unused code:
+...
+tidy check (x86_64-apple-darwin)
+* 470 error codes
+* highest error code: E0591
+...
+
+Here we see the highest error code in use is E0591, so we probably want
+E0592. To be sure, run rg E0592 and check, you should see no references.
+Next, open src/{crate}/diagnostics.rs within the crate where you wish to issue
+the error (e.g., src/librustc_typeck/diagnostics.rs). Ideally, you will add
+the code (in its proper numerical order) into the register_long_diagnostics!
+macro, sort of like this:
+
+#![allow(unused_variables)]
+fn main() {
+register_long_diagnostics! {
+ ...
+ E0592: r##"
+Your extended error text goes here!
+"##,
+}
+}
+
+But you can also add it without an extended description:
+
+#![allow(unused_variables)]
+fn main() {
+register_diagnostics! {
+ ...
+ E0592, // put a description here
+}
+}
+
+To actually issue the error, you can use the struct_span_err! macro:
+
+#![allow(unused_variables)]
+fn main() {
+struct_span_err!(self.tcx.sess, // some path to the session here
+ span, // whatever span in the source you want
+ E0592, // your new error code
+ &format!("text of the error"))
+ .emit() // actually issue the error
+}
+
+If you want to add notes or other snippets, you can invoke methods before you
+call .emit():
+
+#![allow(unused_variables)]
+fn main() {
+struct_span_err!(...)
+ .span_label(another_span, "something to label in the source")
+ .span_note(another_span, "some separate note, probably avoid these")
+ .emit_()
+}
+
+
+The ICE-breaker groups are an easy way to help out with rustc in a
+"piece-meal" fashion, without committing to a larger project.
+ICE-breaker groups are easy to join (just submit a PR!)
+and joining does not entail any particular commitment.
+Once you join an ICE-breaker group, you will be added to
+a list that receives pings on github whenever a new issue is found
+that fits the ICE-breaker group's criteria. If you are interested, you
+can then claim the issue and start working on it.
+Of course, you don't have to wait for new issues to be tagged! If you
+prefer, you can use the Github label for an ICE-breaker group to
+search for existing issues that haven't been claimed yet.
+
+"ICE-breaker issues" are intended to be isolated bugs of middle
+priority:
+
+- By isolated, we mean that we do not expect large-scale refactoring
+to be required to fix the bug.
+- By middle priority, we mean that we'd like to see the bug fixed,
+but it's not such a burning problem that we are dropping everything
+else to fix it. The danger with such bugs, of course, is that they
+can accumulate over time, and the role of the ICE-breaker groups is
+to try and stop that from happening!
+
+
+
+To join an ICE-breaker group, you just have to open a PR adding your
+Github username to the appropriate file in the Rust team repository.
+See the "example PRs" below to get a precise idea and to identify the
+file to edit.
+Also, if you are not already a member of a Rust team then -- in addition
+to adding your name to the file -- you have to checkout the repository and
+run the following command:
+cargo run add-person $your_user_name
+
+Example PRs:
+
+
+To tag an issue as appropriate for an ICE-breaker group, you give
+rustbot a ping command with the name of the ICE-breakers
+team. For example:
+@rustbot ping icebreakers-llvm
+@rustbot ping icebreakers-cleanup-crew
+
+To make these commands shorter and easier to remember, there are aliases,
+defined in the triagebot.toml file. For example:
+@rustbot ping llvm
+@rustbot ping cleanup
+
+Keep in mind that these aliases are meant to make humans' life easier.
+They might be subject to change. If you need to ensure that a command
+will always be valid, prefer the full invocations over the aliases.
+Note though that this should only be done by compiler team members
+or contributors, and is typically done as part of compiler team
+triage.
+
+Github Label: ICEBreaker-Cleanup-Crew
+The "Cleanup Crew" are focused on improving bug reports. Specifically,
+the goal is to try to ensure that every bug report has all the
+information that will be needed for someone to fix it:
+
+- a minimal, standalone example that shows the problem
+- links to duplicates or related bugs
+- if the bug is a regression (something that used to work, but no longer does),
+then a bisection to the PR or nightly that caused the regression
+
+This kind of cleanup is invaluable in getting bugs fixed. Better
+still, it can be done by anybody who knows Rust, without any
+particularly deep knowledge of the compiler.
+Let's look a bit at the workflow for doing "cleanup crew" actions.
+
+Here the ultimate goal is to produce an example that reproduces the same
+problem but without relying on any external crates. Such a test ought to contain
+as little code as possible, as well. This will make it much easier to isolate the problem.
+However, even if the "ultimate minimal test" cannot be achieved, it's
+still useful to post incremental minimizations. For example, if you
+can eliminate some of the external dependencies, that is helpful, and
+so forth.
+It's particularly useful to reduce to an example that works
+in the Rust playground, rather than
+requiring people to checkout a cargo build.
+There are many resources for how to produce minimized test cases. Here
+are a few:
+
+- The rust-reduce tool can try to reduce
+code automatically.
+
+- The C-reduce tool also works
+on Rust code, though it requires that you start from a single
+file. (XXX link to some post explaining how to do it?)
+
+
+- pnkfelix's Rust Bug Minimization Patterns blog post
+
+- This post focuses on "heavy bore" techniques, where you are
+starting with a large, complex cargo project that you wish to
+narrow down to something standalone.
+
+
+
+
+If you are on the "Cleanup Crew", you will sometimes see multiple bug
+reports that seem very similar. You can link one to the other just by
+mentioning the other bug number in a Github comment. Sometimes it is
+useful to close duplicate bugs. But if you do so, you should always
+copy any test case from the bug you are closing to the other bug that
+remains open, as sometimes duplicate-looking bugs will expose
+different facets of the same problem.
+
+For regressions (something that used to work, but no longer does), it
+is super useful if we can figure out precisely when the code stopped
+working. The gold standard is to be able to identify the precise
+PR that broke the code, so we can ping the author, but even
+narrowing it down to a nightly build is helpful, especially as that
+then gives us a range of PRs. (One other challenge is that we
+sometimes land "rollup" PRs, which combine multiple PRs into one.)
+
+To help in figuring out the cause of a regression we have a tool
+called cargo-bisect-rustc. It will automatically download and test
+various builds of rustc. For recent regressions, it is even able to
+use the builds from our CI to track down the regression to a specific
+PR; for older regressions, it will simply identify a nightly.
+To learn to use cargo-bisect-rustc, check out this blog
+post, which gives a quick introduction to how it works. You
+can also ask questions at the Zulip stream
+#t-compiler/cargo-bisect-rustc, or help in improving the tool.
+
+If the regression occurred more than 90 days ago, then
+cargo-bisect-rustc will not able to identify the particular PR that
+caused the regression, just the nightly build. In that case, we can
+identify the set of PRs that this corresponds to by using the git
+history.
+The command rustc +nightly -vV will cause rustc to output a number
+of useful bits of version info, including the commit-hash. Given the
+commit-hash of two nightly versions, you can find all of PRs that have
+landed in between by taking the following steps:
+
+- Go to an update checkout of the rust-lang/rust repository
+- Execute the command
git log --author=bors --format=oneline SHA1..SHA2
+
+
+- This will list out all of the commits by bors, which is our merge bot
+- Each commit corresponds to one PR, and information about the PR should be in the description
+
+
+- Copy and paste that information into the bug report
+
+Often, just eye-balling the PR descriptions (which are included in the
+commit messages) will give you a good idea which one likely caused the
+problem. But if you're unsure feel free to just ping the compiler team
+(@rust-lang/compiler) or else to ping the authors of the PR
+themselves.
+
+Github Label: ICEBreaker-LLVM
+The "LLVM ICE-breakers" are focused on bugs that center around LLVM.
+These bugs often arise because of LLVM optimizations gone awry, or as
+the result of an LLVM upgrade. The goal here is:
+
+- to determine whether the bug is a result of us generating invalid LLVM IR,
+or LLVM misoptimizing;
+- if the former, to fix our IR;
+- if the latter, to try and file a bug on LLVM (or identify an existing bug).
+
+
+The "Debugging LLVM" section of the
+rustc-dev-guide gives a step-by-step process for how to help debug bugs
+caused by LLVM. In particular, it discusses how to emit LLVM IR, run
+the LLVM IR optimization pipeliness, and so forth. You may also find
+it useful to look at the various codegen options listed under -Chelp
+and the internal options under -Zhelp -- there are a number that
+pertain to LLVM (just search for LLVM).
+
+The "Debugging LLVM" section also describes what to do once
+you've identified the bug.
+
+The rustc compiler source and standard library are dual licensed under the Apache License v2.0 and the MIT License unless otherwise specified.
+Detailed licensing information is available in the COPYRIGHT document of the rust-lang/rust repository.
+
+The remaining parts of this guide discuss how the compiler works. They go
+through everything from high-level structure of the compiler to how each stage
+of compilation works. They should be friendly to both readers interested in the
+end-to-end process of compilation and readers interested in learning about a
+specific system they wish to contribute to. If anything is unclear, feel free
+to file an issue on the rustc-dev-guide
+repo or contact the compiler
+team, as detailed in this chapter from Part 1.
+In this part, we will specifically look at the high-level architecture of the
+compiler. Specifically, will look at the query system, incremental compilation,
+and interning. These are three overarching design choices that impact the whole
+compiler.
+
+This chapter is about the overall process of compiling a program -- how
+everything fits together.
+The rust compiler is special in two ways: it does things to your code that
+other compilers don't do (e.g. borrow checking) and it has a lot of
+unconventional implementation choices (e.g. queries). We will talk about these
+in turn in this chapter, and in the rest of the guide, we will look at all the
+individual pieces in more detail.
+
+So first, let's look at what the compiler does to your code. For now, we will
+avoid mentioning how the compiler implements these steps except as needed;
+we'll talk about that later.
+
+- The compile process begins when a user writes a Rust source program in text
+and invokes the
rustc compiler on it. The work that the compiler needs to
+perform is defined by command-line options. For example, it is possible to
+enable nightly features (-Z flags), perform check-only builds, or emit
+LLVM-IR rather than executable machine code. The rustc executable call may
+be indirect through the use of cargo.
+- Command line argument parsing occurs in the
librustc_driver. This crate
+defines the compile configuration that is requested by the user and passes it
+to the rest of the compilation process as a rustc_interface::Config.
+- The raw Rust source text is analyzed by a low-level lexer located in
+
librustc_lexer. At this stage, the source text is turned into a stream of
+atomic source code units known as tokens. The lexer supports the
+Unicode character encoding.
+- The token stream passes through a higher-level lexer located in
+
librustc_parse to prepare for the next stage of the compile process. The
+StringReader struct is used at this stage to perform a set of validations
+and turn strings into interned symbols (interning is discussed later).
+- The lexer has a small interface and doesn't depend directly on the
+diagnostic infrastructure in
rustc. Instead it provides diagnostics as plain
+data which are emitted in librustc_parse::lexer::mod as real diagnostics.
+- The lexer preserves full fidelity information for both IDEs and proc macros.
+- The parser translates the token stream from the lexer into an Abstract Syntax
+Tree (AST). It uses a recursive descent (top-down) approach to syntax
+analysis. The crate entry points for the parser are the
Parser.parse_crate_mod() and
+Parser.parse_mod() methods found in librustc_parse::parser::item. The external
+module parsing entry point is librustc_expand::module::parse_external_mod. And
+the macro parser entry point is rustc_expand::mbe::macro_parser::parse_nt.
+- Parsing is performed with a set of
Parser utility methods including fn bump,
+fn check, fn eat, fn expect, fn look_ahead.
+- Parsing is organized by the semantic construct that is being parsed. Separate
+
parse_* methods can be found in librustc_parse parser directory. The source
+file name follows the construct name. For example, the following files are found
+in the parser:
+
+expr.rspat.rsty.rsstmt.rs
+
+- This naming scheme is used across many compiler stages. You will find
+either a file or directory with the same name across the parsing, lowering,
+type checking, HAIR lowering, and MIR building sources.
+- Macro expansion, AST validation, name resolution, and early linting takes place
+during this stage of the compile process.
+- The parser uses the standard
DiagnosticBuilder API for error handling, but we
+try to recover, parsing a superset of Rust's grammar, while also emitting an error.
+rustc_ast::ast::{Crate, Mod, Expr, Pat, ...} AST nodes are returned from the parser.- We then take the AST and convert it to High-Level Intermediate
+Representation (HIR). This is a compiler-friendly representation of the
+AST. This involves a lot of desugaring of things like loops and
async fn.
+- We use the HIR to do type inference. This is the process of automatic
+detection of the type of an expression.
+- TODO: Maybe some other things are done here? I think initial type checking
+happens here? And trait solving?
+- The HIR is then lowered to Mid-Level Intermediate Representation (MIR).
+
+- Along the way, we construct the HAIR, which is an even more desugared HIR.
+HAIR is used for pattern and exhaustiveness checking. It is also more
+convenient to convert into MIR than HIR is.
+
+
+- The MIR is used for borrow checking.
+- We (want to) do many optimizations on the MIR because it is still
+generic and that improves the code we generate later, improving compilation
+speed too.
+
+- MIR is a higher level (and generic) representation, so it is easier to do
+some optimizations at MIR level than at LLVM-IR level. For example LLVM
+doesn't seem to be able to optimize the pattern the
simplify_try mir
+opt looks for.
+
+
+- Rust code is monomorphized, which means making copies of all the generic
+code with the type parameters replaced by concrete types. To do
+this, we need to collect a list of what concrete types to generate code for.
+This is called monomorphization collection.
+- We then begin what is vaguely called code generation or codegen.
+
+- The code generation stage (codegen) is when higher level
+representations of source are turned into an executable binary.
rustc
+uses LLVM for code generation. The first step is the MIR is then
+converted to LLVM Intermediate Representation (LLVM IR). This is where
+the MIR is actually monomorphized, according to the list we created in
+the previous step.
+- The LLVM IR is passed to LLVM, which does a lot more optimizations on it.
+It then emits machine code. It is basically assembly code with additional
+low-level types and annotations added. (e.g. an ELF object or wasm).
+- The different libraries/binaries are linked together to produce the final
+binary.
+
+
+
+
+Ok, so now that we have a high-level view of what the compiler does to your
+code, let's take a high-level view of how it does all that stuff. There are a
+lot of constraints and conflicting goals that the compiler needs to
+satisfy/optimize for. For example,
+
+- Compilation speed: how fast is it to compile a program. More/better
+compile-time analyses often means compilation is slower.
+
+- Also, we want to support incremental compilation, so we need to take that
+into account. How can we keep track of what work needs to be redone and
+what can be reused if the user modifies their program?
+
+- Also we can't store too much stuff in the incremental cache because
+it would take a long time to load from disk and it could take a lot
+of space on the user's system...
+
+
+
+
+- Compiler memory usage: while compiling a program, we don't want to use more
+memory than we need.
+- Program speed: how fast is your compiled program. More/better compile-time
+analyses often means the compiler can do better optimizations.
+- Program size: how large is the compiled binary? Similar to the previous
+point.
+- Compiler compilation speed: how long does it take to compile the compiler?
+This impacts contributors and compiler maintenance.
+- Implementation complexity: building a compiler is one of the hardest
+things a person/group can do, and Rust is not a very simple language, so how
+do we make the compiler's code base manageable?
+- Compiler correctness: the binaries produced by the compiler should do what
+the input programs says they do, and should continue to do so despite the
+tremendous amount of change constantly going on.
+- Integration: a number of other tools need to use the compiler in
+various ways (e.g. cargo, clippy, miri, RLS) that must be supported.
+- Compiler stability: the compiler should not crash or fail ungracefully on the
+stable channel.
+- Rust stability: the compiler must respect Rust's stability guarantees by not
+breaking programs that previously compiled despite the many changes that are
+always going on to its implementation.
+- Limitations of other tools: rustc uses LLVM in its backend, and LLVM has some
+strengths we leverage and some limitations/weaknesses we need to work around.
+
+So, as you read through the rest of the guide, keep these things in mind. They
+will often inform decisions that we make.
+
+Keep in mind that rustc is a real production-quality product.
+As such, it has its fair share of codebase churn and technical debt. A lot of
+the designs discussed throughout this guide are idealized designs that are not
+fully realized yet. And things keep changing so that it is hard to keep this
+guide completely up to date on everything!
+The compiler definitely has rough edges, but because of its design it is able
+to keep up with the requirements above.
+
+As with most compilers, rustc uses some intermediate representations (IRs) to
+facilitate computations. In general, working directly with the source code is
+extremely inconvenient and error-prone. Source code is designed to be human-friendly while at
+the same time being unambiguous, but it's less convenient for doing something
+like, say, type checking.
+Instead most compilers, including rustc, build some sort of IR out of the
+source code which is easier to analyze. rustc has a few IRs, each optimized
+for different purposes:
+
+- Token stream: the lexer produces a stream of tokens directly from the source
+code. This stream of tokens is easier for the parser to deal with than raw
+text.
+- Abstract Syntax Tree (AST): the abstract syntax tree is built from the stream
+of tokens produced by the lexer. It represents
+pretty much exactly what the user wrote. It helps to do some syntactic sanity
+checking (e.g. checking that a type is expected where the user wrote one).
+- High-level IR (HIR): This is a sort of desugared AST. It's still close
+to what the user wrote syntactically, but it includes some implicit things
+such as some elided lifetimes, etc. This IR is amenable to type checking.
+- HAIR: This is an intermediate between HIR and MIR. It is like the HIR but it
+is fully typed and a bit more desugared (e.g. method calls and implicit
+dereferences are made fully explicit). Moreover, it is easier to lower to MIR
+from HAIR than from HIR.
+- Middle-level IR (MIR): This IR is basically a Control-Flow Graph (CFG). A CFG
+is a type of diagram that shows the basic blocks of a program and how control
+flow can go between them. Likewise, MIR also has a bunch of basic blocks with
+simple typed statements inside them (e.g. assignment, simple computations,
+etc) and control flow edges to other basic blocks (e.g., calls, dropping
+values). MIR is used for borrow checking and other
+important dataflow-based checks, such as checking for uninitialized values.
+It is also used for a series of optimizations and for constant evaluation (via
+MIRI). Because MIR is still generic, we can do a lot of analyses here more
+efficiently than after monomorphization.
+- LLVM IR: This is the standard form of all input to the LLVM compiler. LLVM IR
+is a sort of typed assembly language with lots of annotations. It's
+a standard format that is used by all compilers that use LLVM (e.g. the clang
+C compiler also outputs LLVM IR). LLVM IR is designed to be easy for other
+compilers to emit and also rich enough for LLVM to run a bunch of
+optimizations on it.
+
+One other thing to note is that many values in the compiler are interned.
+This is a performance and memory optimization in which we allocate the values
+in a special allocator called an arena. Then, we pass around references to
+the values allocated in the arena. This allows us to make sure that identical
+values (e.g. types in your program) are only allocated once and can be compared
+cheaply by comparing pointers. Many of the intermediate representations are
+interned.
+
+The first big implementation choice is the query system. The rust compiler
+uses a query system which is unlike most textbook compilers, which are
+organized as a series of passes over the code that execute sequentially. The
+compiler does this to make incremental compilation possible -- that is, if the
+user makes a change to their program and recompiles, we want to do as little
+redundant work as possible to produce the new binary.
+In rustc, all the major steps above are organized as a bunch of queries that
+call each other. For example, there is a query to ask for the type of something
+and another to ask for the optimized MIR of a function. These
+queries can call each other and are all tracked through the query system.
+The results of the queries are cached on disk so that we can tell which
+queries' results changed from the last compilation and only redo those. This is
+how incremental compilation works.
+In principle, for the query-fied steps, we do each of the above for each item
+individually. For example, we will take the HIR for a function and use queries
+to ask for the LLVM IR for that HIR. This drives the generation of optimized
+MIR, which drives the borrow checker, which drives the generation of MIR, and
+so on.
+... except that this is very over-simplified. In fact, some queries are not
+cached on disk, and some parts of the compiler have to run for all code anyway
+for correctness even if the code is dead code (e.g. the borrow checker). For
+example, currently the mir_borrowck query is first executed on all functions
+of a crate. Then the codegen backend invokes the
+collect_and_partition_mono_items query, which first recursively requests the
+optimized_mir for all reachable functions, which in turn runs mir_borrowck
+for that function and then creates codegen units. This kind of split will need
+to remain to ensure that unreachable functions still have their errors emitted.
+Moreover, the compiler wasn't originally built to use a query system; the query
+system has been retrofitted into the compiler, so parts of it are not
+query-fied yet. Also, LLVM isn't our code, so that isn't querified
+either. The plan is to eventually query-fy all of the steps listed in the
+previous section, but as of this writing, only the steps between HIR and
+LLVM-IR are query-fied. That is, lexing and parsing are done all at once for
+the whole program.
+One other thing to mention here is the all-important "typing context",
+TyCtxt, which is a giant struct that is at the center of all things.
+(Note that the name is mostly historic. This is not a "typing context" in the
+sense of Γ or Δ from type theory. The name is retained because that's what
+the name of the struct is in the source code.) All
+queries are defined as methods on the TyCtxt type, and the in-memory query
+cache is stored there too. In the code, there is usually a variable called
+tcx which is a handle on the typing context. You will also see lifetimes with
+the name 'tcx, which means that something is tied to the lifetime of the
+TyCtxt (usually it is stored or interned there).
+
+Types are really important in Rust, and they form the core of a lot of compiler
+analyses. The main type (in the compiler) that represents types (in the user's
+program) is rustc::ty::Ty. This is so important that we have a whole chapter
+on ty::Ty, but for now, we just want to mention that it exists and is the way
+rustc represents types!
+Also note that the rustc::ty module defines the TyCtxt struct we mentioned before.
+
+Compiler performance is a problem that we would like to improve on
+(and are always working on). One aspect of that is parallelizing
+rustc itself.
+Currently, there is only one part of rustc that is already parallel: codegen.
+During monomorphization, the compiler will split up all the code to be
+generated into smaller chunks called codegen units. These are then generated
+by independent instances of LLVM. Since they are independent, we can run them
+in parallel. At the end, the linker is run to combine all the codegen units
+together into one binary.
+However, the rest of the compiler is still not yet parallel. There have been
+lots of efforts spent on this, but it is generally a hard problem. The current
+approach is to turn RefCells into Mutexs -- that is, we
+switch to thread-safe internal mutability. However, there are ongoing
+challenges with lock contention, maintaining query-system invariants under
+concurrency, and the complexity of the code base. One can try out the current
+work by enabling parallel compilation in config.toml. It's still early days,
+but there are already some promising performance improvements.
+
+rustc itself is written in Rust. So how do we compile the compiler? We use an
+older compiler to compile the newer compiler. This is called bootstrapping.
+Bootstrapping has a lot of interesting implications. For example, it means that
+one of the major users of Rust is Rust, so we are constantly testing our own
+software ("eating our own dogfood"). Also, it means building the compiler can
+take a long time because one must first build the new compiler with an older
+compiler and then use that to build the new compiler with itself (sometimes you
+can get away without the full 2-stage build, but for release artifacts you need
+the 2-stage build).
+Bootstrapping also has implications for when features are usable in the
+compiler itself. The build system uses the current beta compiler to build the
+stage-1 bootstrapping compiler. This means that the compiler source code can't
+use some features until they reach beta (because otherwise the beta compiler
+doesn't support them). On the other hand, for compiler intrinsics
+and internal features, we may be able to use them immediately because the
+stage-1 bootstrapping compiler will support them.
+
+
+- Does LLVM ever do optimizations in debug builds?
+- How do I explore phases of the compile process in my own sources (lexer,
+parser, HIR, etc)? - e.g.,
cargo rustc -- -Zunpretty=hir-tree allows you to
+view HIR representation
+- What is the main source entry point for
X?
+- Where do phases diverge for cross-compilation to machine code across
+different platforms?
+
+
+
+- Command line parsing
+
+
+- Lexical Analysis: Lex the user program to a stream of tokens
+
+
+- Parsing: Parse the stream of tokens to an Abstract Syntax Tree (AST)
+
+
+- The High Level Intermediate Representation (HIR)
+
+
+- Type Inference
+
+
+- The Mid Level Intermediate Representation (MIR)
+
+
+- The Borrow Checker
+
+
+- MIR Optimizations
+
+
+- Code Generation
+
+
+
+
+
+The main Rust repository consists of a src directory, under which
+there live many crates. These crates contain the sources for the
+standard library and the compiler. This document, of course, focuses
+on the latter.
+Rustc consists of a number of crates, including rustc_ast,
+rustc, rustc_target, rustc_codegen, rustc_driver, and
+many more. The source for each crate can be found in a directory
+like src/libXXX, where XXX is the crate name.
+(N.B. The names and divisions of these crates are not set in
+stone and may change over time. For the time being, we tend towards a
+finer-grained division to help with compilation time, though as incremental
+compilation improves, that may change.)
+The dependency structure of these crates is roughly a diamond:
+ rustc_driver
+ / | \
+ / | \
+ / | \
+ / v \
+rustc_codegen rustc_borrowck ... rustc_metadata
+ \ | /
+ \ | /
+ \ | /
+ \ v /
+ rustc_middle
+ |
+ v
+ rustc_ast
+ / \
+ / \
+ rustc_span rustc_builtin_macros
+
+The rustc_driver crate, at the top of this lattice, is effectively
+the "main" function for the rust compiler. It doesn't have much "real
+code", but instead ties together all of the code defined in the other
+crates and defines the overall flow of execution. (As we transition
+more and more to the query model, however, the
+"flow" of compilation is becoming less centrally defined.)
+At the other extreme, the rustc_middle crate defines the common and
+pervasive data structures that all the rest of the compiler uses
+(e.g. how to represent types, traits, and the program itself). It
+also contains some amount of the compiler itself, although that is
+relatively limited.
+Finally, all the crates in the bulge in the middle define the bulk of
+the compiler – they all depend on rustc_middle, so that they can make use
+of the various types defined there, and they export public routines
+that rustc_driver will invoke as needed (more and more, what these
+crates export are "query definitions", but those are covered later
+on).
+Below rustc_middle lie various crates that make up the parser and error
+reporting mechanism. They are also an internal part
+of the compiler and not intended to be stable (though they do wind up
+getting used by some crates in the wild; a practice we hope to
+gradually phase out).
+
+The Rust compiler is in a bit of transition right now. It used to be a
+purely "pass-based" compiler, where we ran a number of passes over the
+entire program, and each did a particular check of transformation. We
+are gradually replacing this pass-based code with an alternative setup
+based on on-demand queries. In the query-model, we work backwards,
+executing a query that expresses our ultimate goal (e.g. "compile
+this crate"). This query in turn may make other queries (e.g. "get me
+a list of all modules in the crate"). Those queries make other queries
+that ultimately bottom out in the base operations, like parsing the
+input, running the type-checker, and so forth. This on-demand model
+permits us to do exciting things like only do the minimal amount of
+work needed to type-check a single function. It also helps with
+incremental compilation. (For details on defining queries, check out
+the query model.)
+Regardless of the general setup, the basic operations that the
+compiler must perform are the same. The only thing that changes is
+whether these operations are invoked front-to-back, or on demand. In
+order to compile a Rust crate, these are the general steps that we
+take:
+
+- Parsing input
+
+- this processes the
.rs files and produces the AST
+("abstract syntax tree")
+- the AST is defined in
src/librustc_ast/ast.rs. It is intended to match the lexical
+syntax of the Rust language quite closely.
+
+
+- Name resolution, macro expansion, and configuration
+
+- once parsing is complete, we process the AST recursively, resolving
+paths and expanding macros. This same process also processes
#[cfg]
+nodes, and hence may strip things out of the AST as well.
+
+
+- Lowering to HIR
+
+- Once name resolution completes, we convert the AST into the HIR,
+or "high-level intermediate representation". The HIR is defined in
+
src/librustc_middle/hir/; that module also includes the lowering code.
+- The HIR is a lightly desugared variant of the AST. It is more processed
+than the AST and more suitable for the analyses that follow.
+It is not required to match the syntax of the Rust language.
+- As a simple example, in the AST, we preserve the parentheses
+that the user wrote, so
((1 + 2) + 3) and 1 + 2 + 3 parse
+into distinct trees, even though they are equivalent. In the
+HIR, however, parentheses nodes are removed, and those two
+expressions are represented in the same way.
+
+
+- Type-checking and subsequent analyses
+
+- An important step in processing the HIR is to perform type
+checking. This process assigns types to every HIR expression,
+for example, and also is responsible for resolving some
+"type-dependent" paths, such as field accesses (
x.f – we
+can't know what field f is being accessed until we know the
+type of x) and associated type references (T::Item – we
+can't know what type Item is until we know what T is).
+- Type checking creates "side-tables" (
TypeckTables) that include
+the types of expressions, the way to resolve methods, and so forth.
+- After type-checking, we can do other analyses, such as privacy checking.
+
+
+- Lowering to MIR and post-processing
+
+- Once type-checking is done, we can lower the HIR into MIR ("middle IR"),
+which is a very desugared version of Rust, well suited to borrowck
+but also to certain high-level optimizations.
+
+
+- Translation to LLVM and LLVM optimizations
+
+- From MIR, we can produce LLVM IR.
+- LLVM then runs its various optimizations, which produces a number of
+
.o files (one for each "codegen unit").
+
+
+- Linking
+
+- Finally, those
.o files are linked together.
+
+
+
+
+As described in the high-level overview of the compiler, the
+Rust compiler is currently transitioning from a traditional "pass-based"
+setup to a "demand-driven" system. The Compiler Query System is the
+key to our new demand-driven organization. The idea is pretty
+simple. You have various queries that compute things about the input
+– for example, there is a query called type_of(def_id) that, given
+the def-id of some item, will compute the type of that item and return
+it to you.
+Query execution is memoized – so the first time you invoke a
+query, it will go do the computation, but the next time, the result is
+returned from a hashtable. Moreover, query execution fits nicely into
+incremental computation; the idea is roughly that, when you do a
+query, the result may be returned to you by loading stored data
+from disk (but that's a separate topic we won't discuss further here).
+The overall vision is that, eventually, the entire compiler
+control-flow will be query driven. There will effectively be one
+top-level query ("compile") that will run compilation on a crate; this
+will in turn demand information about that crate, starting from the
+end. For example:
+
+- This "compile" query might demand to get a list of codegen-units
+(i.e. modules that need to be compiled by LLVM).
+- But computing the list of codegen-units would invoke some subquery
+that returns the list of all modules defined in the Rust source.
+- That query in turn would invoke something asking for the HIR.
+- This keeps going further and further back until we wind up doing the
+actual parsing.
+
+However, that vision is not fully realized. Still, big chunks of the
+compiler (for example, generating MIR) work exactly like this.
+
+The Incremental Compilation in Detail chapter gives a more
+in-depth description of what queries are and how they work.
+If you intend to write a query of your own, this is a good read.
+
+To invoke a query is simple. The tcx ("type context") offers a method
+for each defined query. So, for example, to invoke the type_of
+query, you would just do this:
+let ty = tcx.type_of(some_def_id);
+
+
+So you may be wondering what happens when you invoke a query
+method. The answer is that, for each query, the compiler maintains a
+cache – if your query has already been executed, then, the answer is
+simple: we clone the return value out of the cache and return it
+(therefore, you should try to ensure that the return types of queries
+are cheaply cloneable; insert a Rc if necessary).
+
+If, however, the query is not in the cache, then the compiler will
+try to find a suitable provider. A provider is a function that has
+been defined and linked into the compiler somewhere that contains the
+code to compute the result of the query.
+Providers are defined per-crate. The compiler maintains,
+internally, a table of providers for every crate, at least
+conceptually. Right now, there are really two sets: the providers for
+queries about the local crate (that is, the one being compiled)
+and providers for queries about external crates (that is,
+dependencies of the local crate). Note that what determines the crate
+that a query is targeting is not the kind of query, but the key.
+For example, when you invoke tcx.type_of(def_id), that could be a
+local query or an external query, depending on what crate the def_id
+is referring to (see the self::keys::Key trait for more
+information on how that works).
+Providers always have the same signature:
+fn provider<'tcx>(
+ tcx: TyCtxt<'tcx>,
+ key: QUERY_KEY,
+) -> QUERY_RESULT {
+ ...
+}
+
+Providers take two arguments: the tcx and the query key.
+They return the result of the query.
+
+When the tcx is created, it is given the providers by its creator using
+the Providers struct. This struct is generated by
+the macros here, but it is basically a big list of function pointers:
+struct Providers {
+ type_of: for<'tcx> fn(TyCtxt<'tcx>, DefId) -> Ty<'tcx>,
+ ...
+}
+
+At present, we have one copy of the struct for local crates, and one
+for external crates, though the plan is that we may eventually have
+one per crate.
+These Providers structs are ultimately created and populated by
+librustc_driver, but it does this by distributing the work
+throughout the other rustc_* crates. This is done by invoking
+various provide functions. These functions tend to look
+something like this:
+pub fn provide(providers: &mut Providers) {
+ *providers = Providers {
+ type_of,
+ ..*providers
+ };
+}
+
+That is, they take an &mut Providers and mutate it in place. Usually
+we use the formulation above just because it looks nice, but you could
+as well do providers.type_of = type_of, which would be equivalent.
+(Here, type_of would be a top-level function, defined as we saw
+before.) So, if we want to add a provider for some other query,
+let's call it fubar, into the crate above, we might modify the provide()
+function like so:
+pub fn provide(providers: &mut Providers) {
+ *providers = Providers {
+ type_of,
+ fubar,
+ ..*providers
+ };
+}
+
+fn fubar<'tcx>(tcx: TyCtxt<'tcx>, key: DefId) -> Fubar<'tcx> { ... }
+
+N.B. Most of the rustc_* crates only provide local
+providers. Almost all extern providers wind up going through the
+rustc_metadata crate, which loads the information
+from the crate metadata. But in some cases there are crates that
+provide queries for both local and external crates, in which case
+they define both a provide and a
+provide_extern function that rustc_driver
+can invoke.
+
+So suppose you want to add a new kind of query, how do you do so?
+Well, defining a query takes place in two steps:
+
+- first, you have to specify the query name and arguments; and then,
+- you have to supply query providers where needed.
+
+To specify the query name and arguments, you simply add an entry to
+the big macro invocation in
+src/librustc_middle/query/mod.rs, which looks something like:
+rustc_queries! {
+ Other {
+ /// Records the type of every item.
+ query type_of(key: DefId) -> Ty<'tcx> {
+ cache { key.is_local() }
+ }
+ }
+
+ ...
+}
+
+Queries are grouped into categories (Other, Codegen, TypeChecking, etc.).
+Each group contains one or more queries. Each query definition is broken up like
+this:
+query type_of(key: DefId) -> Ty<'tcx> { ... }
+^^ ^^^^^^^ ^^^^^ ^^^^^^^^ ^^^
+| | | | |
+| | | | query modifiers
+| | | result type of query
+| | query key type
+| name of query
+query keyword
+
+Let's go over them one by one:
+
+- Query keyword: indicates a start of a query definition.
+- Name of query: the name of the query method
+(
tcx.type_of(..)). Also used as the name of a struct
+(ty::queries::type_of) that will be generated to represent
+this query.
+- Query key type: the type of the argument to this query.
+This type must implement the
ty::query::keys::Key trait, which
+defines (for example) how to map it to a crate, and so forth.
+- Result type of query: the type produced by this query. This type
+should (a) not use
RefCell or other interior mutability and (b) be
+cheaply cloneable. Interning or using Rc or Arc is recommended for
+non-trivial data types.
+
+- The one exception to those rules is the
ty::steal::Steal type,
+which is used to cheaply modify MIR in place. See the definition
+of Steal for more details. New uses of Steal should not be
+added without alerting @rust-lang/compiler.
+
+
+- Query modifiers: various flags and options that customize how the
+query is processed (mostly with respect to incremental compilation).
+
+So, to add a query:
+
+- Add an entry to
rustc_queries! using the format above.
+- Link the provider by modifying the appropriate
provide method;
+or add a new one if needed and ensure that rustc_driver is invoking it.
+
+
+For each kind, the rustc_queries macro will generate a "query struct"
+named after the query. This struct is a kind of a place-holder
+describing the query. Each such struct implements the
+self::config::QueryConfig trait, which has associated types for the
+key/value of that particular query. Basically the code generated looks something
+like this:
+// Dummy struct representing a particular kind of query:
+pub struct type_of<'tcx> { data: PhantomData<&'tcx ()> }
+
+impl<'tcx> QueryConfig for type_of<'tcx> {
+ type Key = DefId;
+ type Value = Ty<'tcx>;
+
+ const NAME: QueryName = QueryName::type_of;
+ const CATEGORY: ProfileCategory = ProfileCategory::Other;
+}
+
+There is an additional trait that you may wish to implement called
+self::config::QueryDescription. This trait is
+used during cycle errors to give a "human readable" name for the query,
+so that we can summarize what was happening when the cycle occurred.
+Implementing this trait is optional if the query key is DefId, but
+if you don't implement it, you get a pretty generic error ("processing foo...").
+You can put new impls into the config module. They look something like this:
+impl<'tcx> QueryDescription for queries::type_of<'tcx> {
+ fn describe(tcx: TyCtxt, key: DefId) -> String {
+ format!("computing the type of `{}`", tcx.def_path_str(key))
+ }
+}
+
+Another option is to add desc modifier:
+rustc_queries! {
+ Other {
+ /// Records the type of every item.
+ query type_of(key: DefId) -> Ty<'tcx> {
+ desc { |tcx| "computing the type of `{}`", tcx.def_path_str(key) }
+ }
+ }
+}
+
+rustc_queries macro will generate an appropriate impl automatically.
+
+This chapter provides a deeper dive into the abstract model queries are built on.
+It does not go into implementation details but tries to explain
+the underlying logic. The examples here, therefore, have been stripped down and
+simplified and don't directly reflect the compilers internal APIs.
+
+Abstractly we view the compiler's knowledge about a given crate as a "database"
+and queries are the way of asking the compiler questions about it, i.e.
+we "query" the compiler's "database" for facts.
+However, there's something special to this compiler database: It starts out empty
+and is filled on-demand when queries are executed. Consequently, a query must
+know how to compute its result if the database does not contain it yet. For
+doing so, it can access other queries and certain input values that the database
+is pre-filled with on creation.
+A query thus consists of the following things:
+
+- A name that identifies the query
+- A "key" that specifies what we want to look up
+- A result type that specifies what kind of result it yields
+- A "provider" which is a function that specifies how the result is to be
+computed if it isn't already present in the database.
+
+As an example, the name of the type_of query is type_of, its query key is a
+DefId identifying the item we want to know the type of, the result type is
+Ty<'tcx>, and the provider is a function that, given the query key and access
+to the rest of the database, can compute the type of the item identified by the
+key.
+So in some sense a query is just a function that maps the query key to the
+corresponding result. However, we have to apply some restrictions in order for
+this to be sound:
+
+- The key and result must be immutable values.
+- The provider function must be a pure function, that is, for the same key it
+must always yield the same result.
+- The only parameters a provider function takes are the key and a reference to
+the "query context" (which provides access to rest of the "database").
+
+The database is built up lazily by invoking queries. The query providers will
+invoke other queries, for which the result is either already cached or computed
+by calling another query provider. These query provider invocations
+conceptually form a directed acyclic graph (DAG) at the leaves of which are
+input values that are already known when the query context is created.
+
+Results of query invocations are "memoized" which means that the query context
+will cache the result in an internal table and, when the query is invoked with
+the same query key again, will return the result from the cache instead of
+running the provider again.
+This caching is crucial for making the query engine efficient. Without
+memoization the system would still be sound (that is, it would yield the same
+results) but the same computations would be done over and over again.
+Memoization is one of the main reasons why query providers have to be pure
+functions. If calling a provider function could yield different results for
+each invocation (because it accesses some global mutable state) then we could
+not memoize the result.
+
+When the query context is created, it is still empty: No queries have been
+executed, no results are cached. But the context already provides access to
+"input" data, i.e. pieces of immutable data that were computed before the
+context was created and that queries can access to do their computations.
+Currently this input data consists mainly of the HIR map, upstream crate
+metadata, and the command-line
+options the compiler was invoked with. In the future, inputs will just consist
+of command-line options and a list of source files -- the HIR map will itself
+be provided by a query which processes these source files.
+Without inputs, queries would live in a void without anything to compute their
+result from (remember, query providers only have access to other queries and
+the context but not any other outside state or information).
+For a query provider, input data and results of other queries look exactly the
+same: It just tells the context "give me the value of X". Because input data
+is immutable, the provider can rely on it being the same across
+different query invocations, just as is the case for query results.
+
+How does this DAG of query invocations come into existence? At some point
+the compiler driver will create the, as yet empty, query context. It will then,
+from outside of the query system, invoke the queries it needs to perform its
+task. This looks something like the following:
+fn compile_crate() {
+ let cli_options = ...;
+ let hir_map = ...;
+
+ // Create the query context `tcx`
+ let tcx = TyCtxt::new(cli_options, hir_map);
+
+ // Do type checking by invoking the type check query
+ tcx.type_check_crate();
+}
+
+The type_check_crate query provider would look something like the following:
+fn type_check_crate_provider(tcx, _key: ()) {
+ let list_of_hir_items = tcx.hir_map.list_of_items();
+
+ for item_def_id in list_of_hir_items {
+ tcx.type_check_item(item_def_id);
+ }
+}
+
+We see that the type_check_crate query accesses input data
+(tcx.hir_map.list_of_items()) and invokes other queries
+(type_check_item). The type_check_item
+invocations will themselves access input data and/or invoke other queries,
+so that in the end the DAG of query invocations will be built up backwards
+from the node that was initially executed:
+ (2) (1)
+ list_of_all_hir_items <----------------------------- type_check_crate()
+ |
+ (5) (4) (3) |
+ Hir(foo) <--- type_of(foo) <--- type_check_item(foo) <-------+
+ | |
+ +-----------------+ |
+ | |
+ (7) v (6) (8) |
+ Hir(bar) <--- type_of(bar) <--- type_check_item(bar) <-------+
+
+// (x) denotes invocation order
+
+We also see that often a query result can be read from the cache:
+type_of(bar) was computed for type_check_item(foo) so when
+type_check_item(bar) needs it, it is already in the cache.
+Query results stay cached in the query context as long as the context lives.
+So if the compiler driver invoked another query later on, the above graph
+would still exist and already executed queries would not have to be re-done.
+
+Earlier we stated that query invocations form a DAG. However, it would be easy
+to form a cyclic graph by, for example, having a query provider like the
+following:
+fn cyclic_query_provider(tcx, key) -> u32 {
+ // Invoke the same query with the same key again
+ tcx.cyclic_query(key)
+}
+
+Since query providers are regular functions, this would behave much as expected:
+Evaluation would get stuck in an infinite recursion. A query like this would not
+be very useful either. However, sometimes certain kinds of invalid user input
+can result in queries being called in a cyclic way. The query engine includes
+a check for cyclic invocations and, because cycles are an irrecoverable error,
+will abort execution with a "cycle error" messages that tries to be human
+readable.
+At some point the compiler had a notion of "cycle recovery", that is, one could
+"try" to execute a query and if it ended up causing a cycle, proceed in some
+other fashion. However, this was later removed because it is not entirely
+clear what the theoretical consequences of this are, especially regarding
+incremental compilation.
+
+Some queries have their result wrapped in a Steal<T> struct. These queries
+behave exactly the same as regular with one exception: Their result is expected
+to be "stolen" out of the cache at some point, meaning some other part of the
+program is taking ownership of it and the result cannot be accessed anymore.
+This stealing mechanism exists purely as a performance optimization because some
+result values are too costly to clone (e.g. the MIR of a function). It seems
+like result stealing would violate the condition that query results must be
+immutable (after all we are moving the result value out of the cache) but it is
+OK as long as the mutation is not observable. This is achieved by two things:
+
+- Before a result is stolen, we make sure to eagerly run all queries that
+might ever need to read that result. This has to be done manually by calling
+those queries.
+- Whenever a query tries to access a stolen result, we make the compiler ICE so
+that such a condition cannot go unnoticed.
+
+This is not an ideal setup because of the manual intervention needed, so it
+should be used sparingly and only when it is well known which queries might
+access a given result. In practice, however, stealing has not turned out to be
+much of a maintenance burden.
+To summarize: "Steal queries" break some of the rules in a controlled way.
+There are checks in place that make sure that nothing can go silently wrong.
+
+The query model has some properties that make it actually feasible to evaluate
+multiple queries in parallel without too much of an effort:
+
+- All data a query provider can access is accessed via the query context, so
+the query context can take care of synchronizing access.
+- Query results are required to be immutable so they can safely be used by
+different threads concurrently.
+
+The nightly compiler already implements parallel query evaluation as follows:
+When a query foo is evaluated, the cache table for foo is locked.
+
+- If there already is a result, we can clone it, release the lock and
+we are done.
+- If there is no cache entry and no other active query invocation computing the
+same result, we mark the key as being "in progress", release the lock and
+start evaluating.
+- If there is another query invocation for the same key in progress, we
+release the lock, and just block the thread until the other invocation has
+computed the result we are waiting for. This cannot deadlock because, as
+mentioned before, query invocations form a DAG. Some thread will always make
+progress.
+
+
+The incremental compilation scheme is, in essence, a surprisingly
+simple extension to the overall query system. We'll start by describing
+a slightly simplified variant of the real thing – the "basic algorithm" –
+and then describe some possible improvements.
+
+The basic algorithm is
+called the red-green algorithm. The high-level idea is
+that, after each run of the compiler, we will save the results of all
+the queries that we do, as well as the query DAG. The
+query DAG is a DAG that indexes which queries executed which
+other queries. So, for example, there would be an edge from a query Q1
+to another query Q2 if computing Q1 required computing Q2 (note that
+because queries cannot depend on themselves, this results in a DAG and
+not a general graph).
+On the next run of the compiler, then, we can sometimes reuse these
+query results to avoid re-executing a query. We do this by assigning
+every query a color:
+
+- If a query is colored red, that means that its result during
+this compilation has changed from the previous compilation.
+- If a query is colored green, that means that its result is
+the same as the previous compilation.
+
+There are two key insights here:
+
+- First, if all the inputs to query Q are colored green, then the
+query Q must result in the same value as last time and hence
+need not be re-executed (or else the compiler is not deterministic).
+- Second, even if some inputs to a query changes, it may be that it
+still produces the same result as the previous compilation. In
+particular, the query may only use part of its input.
+
+- Therefore, after executing a query, we always check whether it
+produced the same result as the previous time. If it did, we
+can still mark the query as green, and hence avoid re-executing
+dependent queries.
+
+
+
+
+At the core of incremental compilation is an algorithm called
+"try-mark-green". It has the job of determining the color of a given
+query Q (which must not have yet been executed). In cases where Q has
+red inputs, determining Q's color may involve re-executing Q so that
+we can compare its output, but if all of Q's inputs are green, then we
+can conclude that Q must be green without re-executing it or inspecting
+its value at all. In the compiler, this allows us to avoid
+deserializing the result from disk when we don't need it, and in fact
+enables us to sometimes skip serializing the result as well
+(see the refinements section below).
+Try-mark-green works as follows:
+
+- First check if the query Q was executed during the previous compilation.
+
+- If not, we can just re-execute the query as normal, and assign it the
+color of red.
+
+
+- If yes, then load the 'dependent queries' of Q.
+- If there is a saved result, then we load the
reads(Q) vector from the
+query DAG. The "reads" is the set of queries that Q executed during
+its execution.
+
+- For each query R in
reads(Q), we recursively demand the color
+of R using try-mark-green.
+
+- Note: it is important that we visit each node in
reads(Q) in same order
+as they occurred in the original compilation. See the section on the
+query DAG below.
+- If any of the nodes in
reads(Q) wind up colored red, then Q is
+dirty.
+
+- We re-execute Q and compare the hash of its result to the hash of the
+result from the previous compilation.
+- If the hash has not changed, we can mark Q as green and return.
+
+
+- Otherwise, all of the nodes in
reads(Q) must be green. In that
+case, we can color Q as green and return.
+
+
+
+
+
+
+
+The query DAG code is stored in
+src/librustc_middle/dep_graph. Construction of the DAG is done
+by instrumenting the query execution.
+One key point is that the query DAG also tracks ordering; that is, for
+each query Q, we not only track the queries that Q reads, we track the
+order in which they were read. This allows try-mark-green to walk
+those queries back in the same order. This is important because once a
+subquery comes back as red, we can no longer be sure that Q will continue
+along the same path as before. That is, imagine a query like this:
+fn main_query(tcx) {
+ if tcx.subquery1() {
+ tcx.subquery2()
+ } else {
+ tcx.subquery3()
+ }
+}
+
+Now imagine that in the first compilation, main_query starts by
+executing subquery1, and this returns true. In that case, the next
+query main_query executes will be subquery2, and subquery3 will
+not be executed at all.
+But now imagine that in the next compilation, the input has
+changed such that subquery1 returns false. In this case, subquery2
+would never execute. If try-mark-green were to visit reads(main_query) out
+of order, however, it might visit subquery2 before subquery1, and hence
+execute it.
+This can lead to ICEs and other problems in the compiler.
+
+In the description of the basic algorithm, we said that at the end of
+compilation we would save the results of all the queries that were
+performed. In practice, this can be quite wasteful – many of those
+results are very cheap to recompute, and serializing and deserializing
+them is not a particular win. In practice, what we would do is to save
+the hashes of all the subqueries that we performed. Then, in select cases,
+we also save the results.
+This is why the incremental algorithm separates computing the
+color of a node, which often does not require its value, from
+computing the result of a node. Computing the result is done via a simple
+algorithm like so:
+
+- Check if a saved result for Q is available. If so, compute the color of Q.
+If Q is green, deserialize and return the saved result.
+- Otherwise, execute Q.
+
+- We can then compare the hash of the result and color Q as green if
+it did not change.
+
+
+
+
+The initial design document can be found here, which expands
+on the memoization details, provides more high-level overview and motivation
+for this system.
+
+
+
+The incremental compilation scheme is, in essence, a surprisingly
+simple extension to the overall query system. It relies on the fact that:
+
+- queries are pure functions -- given the same inputs, a query will always
+yield the same result, and
+- the query model structures compilation in an acyclic graph that makes
+dependencies between individual computations explicit.
+
+This chapter will explain how we can use these properties for making things
+incremental and then goes on to discuss version implementation issues.
+
+As explained in the query evaluation model primer, query
+invocations form a directed-acyclic graph. Here's the example from the
+previous chapter again:
+ list_of_all_hir_items <----------------------------- type_check_crate()
+ |
+ |
+ Hir(foo) <--- type_of(foo) <--- type_check_item(foo) <-------+
+ | |
+ +-----------------+ |
+ | |
+ v |
+ Hir(bar) <--- type_of(bar) <--- type_check_item(bar) <-------+
+
+Since every access from one query to another has to go through the query
+context, we can record these accesses and thus actually build this dependency
+graph in memory. With dependency tracking enabled, when compilation is done,
+we know which queries were invoked (the nodes of the graph) and for each
+invocation, which other queries or input has gone into computing the query's
+result (the edges of the graph).
+Now suppose we change the source code of our program so that
+HIR of bar looks different than before. Our goal is to only recompute
+those queries that are actually affected by the change while re-using
+the cached results of all the other queries. Given the dependency graph we can
+do exactly that. For a given query invocation, the graph tells us exactly
+what data has gone into computing its results, we just have to follow the
+edges until we reach something that has changed. If we don't encounter
+anything that has changed, we know that the query still would evaluate to
+the same result we already have in our cache.
+Taking the type_of(foo) invocation from above as an example, we can check
+whether the cached result is still valid by following the edges to its
+inputs. The only edge leads to Hir(foo), an input that has not been affected
+by the change. So we know that the cached result for type_of(foo) is still
+valid.
+The story is a bit different for type_check_item(foo): We again walk the
+edges and already know that type_of(foo) is fine. Then we get to
+type_of(bar) which we have not checked yet, so we walk the edges of
+type_of(bar) and encounter Hir(bar) which has changed. Consequently
+the result of type_of(bar) might yield a different same result than what we
+have in the cache and, transitively, the result of type_check_item(foo)
+might have changed too. We thus re-run type_check_item(foo), which in
+turn will re-run type_of(bar), which will yield an up-to-date result
+because it reads the up-to-date version of Hir(bar).
+
+If you read the previous paragraph carefully you'll notice that it says that
+type_of(bar) might have changed because one of its inputs has changed.
+There's also the possibility that it might still yield exactly the same
+result even though its input has changed. Consider an example with a
+simple query that just computes the sign of an integer:
+ IntValue(x) <---- sign_of(x) <--- some_other_query(x)
+
+Let's say that IntValue(x) starts out as 1000 and then is set to 2000.
+Even though IntValue(x) is different in the two cases, sign_of(x) yields
+the result + in both cases.
+If we follow the basic algorithm, however, some_other_query(x) would have to
+(unnecessarily) be re-evaluated because it transitively depends on a changed
+input. Change detection yields a "false positive" in this case because it has
+to conservatively assume that some_other_query(x) might be affected by that
+changed input.
+Unfortunately it turns out that the actual queries in the compiler are full
+of examples like this and small changes to the input often potentially affect
+very large parts of the output binaries. As a consequence, we had to make the
+change detection system smarter and more accurate.
+
+The "false positives" problem can be solved by interleaving change detection
+and query re-evaluation. Instead of walking the graph all the way to the
+inputs when trying to find out if some cached result is still valid, we can
+check if a result has actually changed after we were forced to re-evaluate
+it.
+We call this algorithm the red-green algorithm because nodes
+in the dependency graph are assigned the color green if we were able to prove
+that its cached result is still valid and the color red if the result has
+turned out to be different after re-evaluating it.
+The meat of red-green change tracking is implemented in the try-mark-green
+algorithm, that, you've guessed it, tries to mark a given node as green:
+fn try_mark_green(tcx, current_node) -> bool {
+
+ // Fetch the inputs to `current_node`, i.e. get the nodes that the direct
+ // edges from `node` lead to.
+ let dependencies = tcx.dep_graph.get_dependencies_of(current_node);
+
+ // Now check all the inputs for changes
+ for dependency in dependencies {
+
+ match tcx.dep_graph.get_node_color(dependency) {
+ Green => {
+ // This input has already been checked before and it has not
+ // changed; so we can go on to check the next one
+ }
+ Red => {
+ // We found an input that has changed. We cannot mark
+ // `current_node` as green without re-running the
+ // corresponding query.
+ return false
+ }
+ Unknown => {
+ // This is the first time we look at this node. Let's try
+ // to mark it green by calling try_mark_green() recursively.
+ if try_mark_green(tcx, dependency) {
+ // We successfully marked the input as green, on to the
+ // next.
+ } else {
+ // We could *not* mark the input as green. This means we
+ // don't know if its value has changed. In order to find
+ // out, we re-run the corresponding query now!
+ tcx.run_query_for(dependency);
+
+ // Fetch and check the node color again. Running the query
+ // has forced it to either red (if it yielded a different
+ // result than we have in the cache) or green (if it
+ // yielded the same result).
+ match tcx.dep_graph.get_node_color(dependency) {
+ Red => {
+ // The input turned out to be red, so we cannot
+ // mark `current_node` as green.
+ return false
+ }
+ Green => {
+ // Re-running the query paid off! The result is the
+ // same as before, so this particular input does
+ // not invalidate `current_node`.
+ }
+ Unknown => {
+ // There is no way a node has no color after
+ // re-running the query.
+ panic!("unreachable")
+ }
+ }
+ }
+ }
+ }
+ }
+
+ // If we have gotten through the entire loop, it means that all inputs
+ // have turned out to be green. If all inputs are unchanged, it means
+ // that the query result corresponding to `current_node` cannot have
+ // changed either.
+ tcx.dep_graph.mark_green(current_node);
+
+ true
+}
+
+// Note: The actual implementation can be found in
+// src/librustc_middle/dep_graph/graph.rs
+
+By using red-green marking we can avoid the devastating cumulative effect of
+having false positives during change detection. Whenever a query is executed
+in incremental mode, we first check if its already green. If not, we run
+try_mark_green() on it. If it still isn't green after that, then we actually
+invoke the query provider to re-compute the result.
+
+The sections above described the underlying algorithm for incremental
+compilation but because the compiler process exits after being finished and
+takes the query context with its result cache with it into oblivion, we have to
+persist data to disk, so the next compilation session can make use of it.
+This comes with a whole new set of implementation challenges:
+
+- The query result cache is stored to disk, so they are not readily available
+for change comparison.
+- A subsequent compilation session will start off with new version of the code
+that has arbitrary changes applied to it. All kinds of IDs and indices that
+are generated from a global, sequential counter (e.g.
NodeId, DefId, etc)
+might have shifted, making the persisted results on disk not immediately
+usable anymore because the same numeric IDs and indices might refer to
+completely new things in the new compilation session.
+- Persisting things to disk comes at a cost, so not every tiny piece of
+information should be actually cached in between compilation sessions.
+Fixed-sized, plain-old-data is preferred to complex things that need to run
+through an expensive (de-)serialization step.
+
+The following sections describe how the compiler currently solves these issues.
+
+As noted before, various IDs (like DefId) are generated by the compiler in a
+way that depends on the contents of the source code being compiled. ID assignment
+is usually deterministic, that is, if the exact same code is compiled twice,
+the same things will end up with the same IDs. However, if something
+changes, e.g. a function is added in the middle of a file, there is no
+guarantee that anything will have the same ID as it had before.
+As a consequence we cannot represent the data in our on-disk cache the same
+way it is represented in memory. For example, if we just stored a piece
+of type information like TyKind::FnDef(DefId, &'tcx Substs<'tcx>) (as we do
+in memory) and then the contained DefId points to a different function in
+a new compilation session we'd be in trouble.
+The solution to this problem is to find "stable" forms for IDs which remain
+valid in between compilation sessions. For the most important case, DefIds,
+these are the so-called DefPaths. Each DefId has a
+corresponding DefPath but in place of a numeric ID, a DefPath is based on
+the path to the identified item, e.g. std::collections::HashMap. The
+advantage of an ID like this is that it is not affected by unrelated changes.
+For example, one can add a new function to std::collections but
+std::collections::HashMap would still be std::collections::HashMap. A
+DefPath is "stable" across changes made to the source code while a DefId
+isn't.
+There is also the DefPathHash which is just a 128-bit hash value of the
+DefPath. The two contain the same information and we mostly use the
+DefPathHash because it simpler to handle, being Copy and self-contained.
+This principle of stable identifiers is used to make the data in the on-disk
+cache resilient to source code changes. Instead of storing a DefId, we store
+the DefPathHash and when we deserialize something from the cache, we map the
+DefPathHash to the corresponding DefId in the current compilation session
+(which is just a simple hash table lookup).
+The HirId, used for identifying HIR components that don't have their own
+DefId, is another such stable ID. It is (conceptually) a pair of a DefPath
+and a LocalId, where the LocalId identifies something (e.g. a hir::Expr)
+locally within its "owner" (e.g. a hir::Item). If the owner is moved around,
+the LocalIds within it are still the same.
+
+In order to do red-green-marking we often need to check if the result of a
+query has changed compared to the result it had during the previous
+compilation session. There are two performance problems with this though:
+
+- We'd like to avoid having to load the previous result from disk just for
+doing the comparison. We already computed the new result and will use that.
+Also loading a result from disk will "pollute" the interners with data that
+is unlikely to ever be used.
+- We don't want to store each and every result in the on-disk cache. For
+example, it would be wasted effort to persist things to disk that are
+already available in upstream crates.
+
+The compiler avoids these problems by using so-called Fingerprints. Each time
+a new query result is computed, the query engine will compute a 128 bit hash
+value of the result. We call this hash value "the Fingerprint of the query
+result". The hashing is (and has to be) done "in a stable way". This means
+that whenever something is hashed that might change in between compilation
+sessions (e.g. a DefId), we instead hash its stable equivalent
+(e.g. the corresponding DefPath). That's what the whole HashStable
+infrastructure is for. This way Fingerprints computed in two
+different compilation sessions are still comparable.
+The next step is to store these fingerprints along with the dependency graph.
+This is cheap since fingerprints are just bytes to be copied. It's also cheap to
+load the entire set of fingerprints together with the dependency graph.
+Now, when red-green-marking reaches the point where it needs to check if a
+result has changed, it can just compare the (already loaded) previous
+fingerprint to the fingerprint of the new result.
+This approach works rather well but it's not without flaws:
+
+-
+
There is a small possibility of hash collisions. That is, two different
+results could have the same fingerprint and the system would erroneously
+assume that the result hasn't changed, leading to a missed update.
+We mitigate this risk by using a high-quality hash function and a 128 bit
+wide hash value. Due to these measures the practical risk of a hash
+collision is negligible.
+
+-
+
Computing fingerprints is quite costly. It is the main reason why incremental
+compilation can be slower than non-incremental compilation. We are forced to
+use a good and thus expensive hash function, and we have to map things to
+their stable equivalents while doing the hashing.
+
+
+
+The initial description of dependency tracking glosses over a few details
+that quickly become a head scratcher when actually trying to implement things.
+In particular it's easy to overlook that we are actually dealing with two
+dependency graphs: The one we built during the previous compilation session and
+the one that we are building for the current compilation session.
+When a compilation session starts, the compiler loads the previous dependency
+graph into memory as an immutable piece of data. Then, when a query is invoked,
+it will first try to mark the corresponding node in the graph as green. This
+means really that we are trying to mark the node in the previous dep-graph
+as green that corresponds to the query key in the current session. How do we
+do this mapping between current query key and previous DepNode? The answer
+is again Fingerprints: Nodes in the dependency graph are identified by a
+fingerprint of the query key. Since fingerprints are stable across compilation
+sessions, computing one in the current session allows us to find a node
+in the dependency graph from the previous session. If we don't find a node with
+the given fingerprint, it means that the query key refers to something that
+did not yet exist in the previous session.
+So, having found the dep-node in the previous dependency graph, we can look
+up its dependencies (i.e. also dep-nodes in the previous graph) and continue with
+the rest of the try-mark-green algorithm. The next interesting thing happens
+when we successfully marked the node as green. At that point we copy the node
+and the edges to its dependencies from the old graph into the new graph. We
+have to do this because the new dep-graph cannot not acquire the
+node and edges via the regular dependency tracking. The tracking system can
+only record edges while actually running a query -- but running the query,
+although we have the result already cached, is exactly what we want to avoid.
+Once the compilation session has finished, all the unchanged parts have been
+copied over from the old into the new dependency graph, while the changed parts
+have been added to the new graph by the tracking system. At this point, the
+new graph is serialized out to disk, alongside the query result cache, and can
+act as the previous dep-graph in a subsequent compilation session.
+
+The system described so far has a somewhat subtle property: If all inputs of a
+dep-node are green then the dep-node itself can be marked as green without
+computing or loading the corresponding query result. Applying this property
+transitively often leads to the situation that some intermediate results are
+never actually loaded from disk, as in the following example:
+ input(A) <-- intermediate_query(B) <-- leaf_query(C)
+
+The compiler might need the value of leaf_query(C) in order to generate some
+output artifact. If it can mark leaf_query(C) as green, it will load the
+result from the on-disk cache. The result of intermediate_query(B) is never
+loaded though. As a consequence, when the compiler persists the new result
+cache by writing all in-memory query results to disk, intermediate_query(B)
+will not be in memory and thus will be missing from the new result cache.
+If there subsequently is another compilation session that actually needs the
+result of intermediate_query(B) it will have to be re-computed even though we
+had a perfectly valid result for it in the cache just before.
+In order to prevent this from happening, the compiler does something called
+"cache promotion": Before emitting the new result cache it will walk all green
+dep-nodes and make sure that their query result is loaded into memory. That way
+the result cache doesn't unnecessarily shrink again.
+
+The compiler backend, the part involving LLVM, is using the query system but
+it is not implemented in terms of queries itself. As a consequence
+it does not automatically partake in dependency tracking. However, the manual
+integration with the tracking system is pretty straight-forward. The compiler
+simply tracks what queries get invoked when generating the initial LLVM version
+of each codegen unit, which results in a dep-node for each of them. In
+subsequent compilation sessions it then tries to mark the dep-node for a CGU as
+green. If it succeeds it knows that the corresponding object and bitcode files
+on disk are still valid. If it doesn't succeed, the entire codegen unit has to
+be recompiled.
+This is the same approach that is used for regular queries. The main differences
+are:
+
+-
+
that we cannot easily compute a fingerprint for LLVM modules (because
+they are opaque C++ objects),
+
+-
+
that the logic for dealing with cached values is rather different from
+regular queries because here we have bitcode and object files instead of
+serialized Rust values in the common result cache file, and
+
+-
+
the operations around LLVM are so expensive in terms of computation time and
+memory consumption that we need to have tight control over what is
+executed when and what stays in memory for how long.
+
+
+The query system could probably be extended with general purpose mechanisms to
+deal with all of the above but so far that seemed like more trouble than it
+would save.
+
+The query system allows for applying modifiers to queries. These
+modifiers affect certain aspects of how the system treats the query with
+respect to incremental compilation:
+
+-
+
eval_always - A query with the eval_always attribute is re-executed
+unconditionally during incremental compilation. I.e. the system will not
+even try to mark the query's dep-node as green. This attribute has two use
+cases:
+
+-
+
eval_always queries can read inputs (from files, global state, etc).
+They can also produce side effects like writing to files and changing global state.
+
+-
+
Some queries are very likely to be re-evaluated because their result
+depends on the entire source code. In this case eval_always can be used
+as an optimization because the system can skip recording dependencies in
+the first place.
+
+
+
+-
+
no_hash - Applying no_hash to a query tells the system to not compute
+the fingerprint of the query's result. This has two consequences:
+
+-
+
Not computing the fingerprint can save quite a bit of time because
+fingerprinting is expensive, especially for large, complex values.
+
+-
+
Without the fingerprint, the system has to unconditionally assume that
+the result of the query has changed. As a consequence anything depending
+on a no_hash query will always be re-executed.
+
+
+
+-
+
cache_on_disk_if - This attribute is what determines which query results
+are persisted in the incremental compilation query result cache. The
+attribute takes an expression that allows per query invocation
+decisions. For example, it makes no sense to store values from upstream
+crates in the cache because they are already available in the upstream
+crate's metadata.
+
+-
+
anon - This attribute makes the system use "anonymous" dep-nodes for the
+given query. An anonymous dep-node is not identified by the corresponding
+query key, instead its ID is computed from the IDs of its dependencies. This
+allows the red-green system to do its change detection even if there is no
+query key available for a given dep-node -- something which is needed for
+handling trait selection because it is not based on queries.
+
+
+
+It's interesting to note that eval_always and no_hash can be used together
+in the so-called "projection query" pattern. It is often the case that there is
+one query that depends on the entirety of the compiler's input (e.g. the indexed HIR)
+and another query that projects individual values out of this monolithic value
+(e.g. a HIR item with a certain DefId). These projection queries allow for
+building change propagation "firewalls" because even if the result of the
+monolithic query changes (which it is very likely to do) the small projections
+can still mostly be marked as green.
+ +------------+
+ | | +---------------+ +--------+
+ | | <---------| projection(x) | <---------| foo(a) |
+ | | +---------------+ +--------+
+ | |
+ | monolithic | +---------------+ +--------+
+ | query | <---------| projection(y) | <---------| bar(b) |
+ | | +---------------+ +--------+
+ | |
+ | | +---------------+ +--------+
+ | | <---------| projection(z) | <---------| baz(c) |
+ | | +---------------+ +--------+
+ +------------+
+
+Let's assume that the result monolithic_query changes so that also the result
+of projection(x) has changed, i.e. both their dep-nodes are being marked as
+red. As a consequence foo(a) needs to be re-executed; but bar(b) and
+baz(c) can be marked as green. However, if foo, bar, and baz would have
+directly depended on monolithic_query then all of them would have had to be
+re-evaluated.
+This pattern works even without eval_always and no_hash but the two
+modifiers can be used to avoid unnecessary overhead. If the monolithic query
+is likely to change at any minor modification of the compiler's input it makes
+sense to mark it as eval_always, thus getting rid of its dependency tracking
+cost. And it always makes sense to mark the monolithic query as no_hash
+because we have the projections to take care of keeping things green as much
+as possible.
+
+There are many things that still can be improved.
+
+The current system is not able to update on-disk caches and the dependency graph
+in-place. Instead it has to rewrite each file entirely in each compilation
+session. The overhead of doing so is a few percent of total compilation time.
+
+Data structures used as query results could be factored in a way that removes
+edges from the dependency graph. Especially "span" information is very volatile,
+so including it in query result will increase the chance that that result won't
+be reusable. See https://github.com/rust-lang/rust/issues/47389 for more
+information.
+
+
+There are various ways to write tests against the dependency graph.
+The simplest mechanisms are the #[rustc_if_this_changed] and
+#[rustc_then_this_would_need] annotations. These are used in compile-fail
+tests to test whether the expected set of paths exist in the dependency graph.
+As an example, see src/test/compile-fail/dep-graph-caller-callee.rs.
+The idea is that you can annotate a test like:
+#[rustc_if_this_changed]
+fn foo() { }
+
+#[rustc_then_this_would_need(TypeckTables)] //~ ERROR OK
+fn bar() { foo(); }
+
+#[rustc_then_this_would_need(TypeckTables)] //~ ERROR no path
+fn baz() { }
+
+This will check whether there is a path in the dependency graph from Hir(foo)
+to TypeckTables(bar). An error is reported for each
+#[rustc_then_this_would_need] annotation that indicates whether a path
+exists. //~ ERROR annotations can then be used to test if a path is found (as
+demonstrated above).
+
+
+The compiler is also capable of dumping the dependency graph for your
+debugging pleasure. To do so, pass the -Z dump-dep-graph flag. The
+graph will be dumped to dep_graph.{txt,dot} in the current
+directory. You can override the filename with the RUST_DEP_GRAPH
+environment variable.
+Frequently, though, the full dep graph is quite overwhelming and not
+particularly helpful. Therefore, the compiler also allows you to filter
+the graph. You can filter in three ways:
+
+- All edges originating in a particular set of nodes (usually a single node).
+- All edges reaching a particular set of nodes.
+- All edges that lie between given start and end nodes.
+
+To filter, use the RUST_DEP_GRAPH_FILTER environment variable, which should
+look like one of the following:
+source_filter // nodes originating from source_filter
+-> target_filter // nodes that can reach target_filter
+source_filter -> target_filter // nodes in between source_filter and target_filter
+
+source_filter and target_filter are a &-separated list of strings.
+A node is considered to match a filter if all of those strings appear in its
+label. So, for example:
+RUST_DEP_GRAPH_FILTER='-> TypeckTables'
+
+would select the predecessors of all TypeckTables nodes. Usually though you
+want the TypeckTables node for some particular fn, so you might write:
+RUST_DEP_GRAPH_FILTER='-> TypeckTables & bar'
+
+This will select only the predecessors of TypeckTables nodes for functions
+with bar in their name.
+Perhaps you are finding that when you change foo you need to re-type-check
+bar, but you don't think you should have to. In that case, you might do:
+RUST_DEP_GRAPH_FILTER='Hir & foo -> TypeckTables & bar'
+
+This will dump out all the nodes that lead from Hir(foo) to
+TypeckTables(bar), from which you can (hopefully) see the source
+of the erroneous edge.
+
+Sometimes, after you dump the dependency graph, you will find some
+path that should not exist, but you will not be quite sure how it came
+to be. When the compiler is built with debug assertions, it can
+help you track that down. Simply set the RUST_FORBID_DEP_GRAPH_EDGE
+environment variable to a filter. Every edge created in the dep-graph
+will be tested against that filter – if it matches, a bug! is
+reported, so you can easily see the backtrace (RUST_BACKTRACE=1).
+The syntax for these filters is the same as described in the previous
+section. However, note that this filter is applied to every edge
+and doesn't handle longer paths in the graph, unlike the previous
+section.
+Example:
+You find that there is a path from the Hir of foo to the type
+check of bar and you don't think there should be. You dump the
+dep-graph as described in the previous section and open dep-graph.txt
+to see something like:
+Hir(foo) -> Collect(bar)
+Collect(bar) -> TypeckTables(bar)
+
+That first edge looks suspicious to you. So you set
+RUST_FORBID_DEP_GRAPH_EDGE to Hir&foo -> Collect&bar, re-run, and
+then observe the backtrace. Voila, bug fixed!
+
+In an effort to support incremental compilation, the latest design of the Rust
+compiler consists of a query-based model.
+The details of this model are (currently) outside the scope of this document,
+however, we explain some background of this model, in an effort
+to explain how we profile its performance. We intend this profiling effort to
+address issue 42678.
+
+
+./configure --enable-debug-assertions
+
+
+Compile the compiler, up to at least stage 1:
+python x.py --stage 1
+
+
+Run the compiler on a source file, supplying two additional debugging flags with
+-Z:
+rustc -Z profile-queries -Z incremental=cache foo.rs
+
+Regarding the two additional parameters:
+
+-Z profile-queries tells the compiler to run a separate thread that profiles
+the queries made by the main compiler thread(s).-Z incremental=cache tells the compiler to "cache" various files that
+describe the compilation dependencies, in the subdirectory cache.
+This command will generate the following files:
+
+profile_queries.html consists of an HTML-based representation of the
+trace of queries.profile_queries.counts.txt consists of a histogram, where each histogram
+"bucket" is a query provider.
+
+
+- This additional flag will add all timed passes to the output files mentioned
+above, in step 2. As described below, these passes appear visually distinct
+from the queries in the HTML output (they currently appear as green boxes, via
+CSS).
+
+
+
+- 4(a). Open the HTML file (
profile_queries.html) with a browser. See
+this section for an explanation of this file.
+- 4(b). Open the data file (
profile_queries.counts.txt) with a text editor, or
+spreadsheet. See this section for an explanation
+of this file.
+
+
+
+The following image gives some example output, from tracing the queries of
+hello_world.rs (a single main function, that prints "hello world" via the
+macro println!). This image only shows a short prefix of the total output; the
+actual output is much longer.
+
+View full HTML output. Note; it could take up
+to a second to properly render depending on your browser.
+Here is the corresponding text output.
+
+The trace of the queries has a formal structure; see
+Trace of Queries for details.
+We style this formal structure as follows:
+
+- Timed passes: Green boxes, when present (via
-Z time-passes), represent
+timed passes in the compiler. In future versions, these passes may be
+replaced by queries, explained below.
+- Labels: Some green and red boxes are labeled with text. Where they are
+present, the labels give the following information:
+
+- The query's provider, sans its key and its result, which
+are often too long to include in these labels.
+- The duration of the provider, as a fraction of the total time (for the
+entire trace). This fraction includes the query's entire extent (that is,
+the sum total of all of its sub-queries).
+
+
+- Query hits: Blue dots represent query hits. They consist of leaves in the
+trace's tree. (CSS class:
hit).
+- Query misses: Red boxes represent query misses. They consist of internal
+nodes in the trace's tree. (CSS class:
miss).
+- Nesting structure: Many red boxes contain nested boxes and dots. This
+nesting structure reflects that some providers depend on results from other
+providers, which consist of their nested children.
+- Some red boxes are labeled with text, and have highlighted borders (light
+red, and bolded). (See heuristics for details).
+
+
+Heuristics-based CSS Classes:
+
+-
+
important -- Trace nodes are important if they have an extent of 6 (or
+more), or they have a duration fraction of one percent (or more). These
+numbers are simple heuristics (currently hard-coded, but easy to modify).
+Important nodes are styled with textual labels, and highlighted borders (light
+red, and bolded).
+
+-
+
frac-50, -40, ... -- Trace nodes whose total duration (self and children)
+take a large fraction of the total duration, at or above 50%, 40%, and so on.
+We style nodes these with larger font and padding.
+
+
+
+The file profile_queries.counts.txt contains a table of information about the
+queries, organized around their providers.
+For each provider (or timed pass, when -Z time-passes is present), we produce:
+
+-
+
A total count --- the total number of times this provider was queried
+
+-
+
A total duration --- the total number of seconds spent running this
+provider, including all providers it may depend on. To get a sense of this
+dependency structure, and inspect a more fine-grained view of these durations,
+see this section.
+
+
+These rows are sorted by total duration, in descending order.
+
+The following example profile_queries.counts.txt file results from running on
+a hello world program (a single main function that uses println to print
+`"hellow world").
+As explained above, the columns consist of provider/pass, count, duration:
+translation,1,0.891
+symbol_name,2658,0.733
+def_symbol_name,2556,0.268
+item_attrs,5566,0.162
+type_of,6922,0.117
+generics_of,8020,0.084
+serialize dep graph,1,0.079
+relevant_trait_impls_for,50,0.063
+def_span,24875,0.061
+expansion,1,0.059
+const checking,1,0.055
+adt_def,1141,0.048
+trait_impls_of,32,0.045
+is_copy_raw,47,0.045
+is_foreign_item,2638,0.042
+fn_sig,2172,0.033
+adt_dtorck_constraint,2,0.023
+impl_trait_ref,2434,0.023
+typeck_tables_of,29,0.022
+item-bodies checking,1,0.017
+typeck_item_bodies,1,0.017
+is_default_impl,2320,0.017
+borrow checking,1,0.014
+borrowck,4,0.014
+mir_validated,4,0.013
+adt_destructor,10,0.012
+layout_raw,258,0.010
+load_dep_graph,1,0.007
+item-types checking,1,0.005
+mir_const,2,0.005
+name resolution,1,0.004
+is_object_safe,35,0.003
+is_sized_raw,89,0.003
+parsing,1,0.003
+is_freeze_raw,11,0.001
+privacy checking,1,0.001
+privacy_access_levels,5,0.001
+resolving dependency formats,1,0.001
+adt_sized_constraint,9,0.001
+wf checking,1,0.001
+liveness checking,1,0.001
+compute_incremental_hashes_map,1,0.001
+match checking,1,0.001
+type collecting,1,0.001
+param_env,31,0.000
+effect checking,1,0.000
+trait_def,140,0.000
+lowering ast -> hir,1,0.000
+predicates_of,70,0.000
+extern_crate,319,0.000
+lifetime resolution,1,0.000
+is_const_fn,6,0.000
+intrinsic checking,1,0.000
+translation item collection,1,0.000
+impl_polarity,15,0.000
+creating allocators,1,0.000
+language item collection,1,0.000
+crate injection,1,0.000
+early lint checks,1,0.000
+indexing hir,1,0.000
+maybe creating a macro crate,1,0.000
+coherence checking,1,0.000
+optimized_mir,6,0.000
+is_panic_runtime,33,0.000
+associated_item_def_ids,7,0.000
+needs_drop_raw,10,0.000
+lint checking,1,0.000
+complete gated feature checking,1,0.000
+stability index,1,0.000
+region_maps,11,0.000
+super_predicates_of,8,0.000
+coherent_trait,2,0.000
+AST validation,1,0.000
+loop checking,1,0.000
+static item recursion checking,1,0.000
+variances_of,11,0.000
+associated_item,5,0.000
+plugin loading,1,0.000
+looking for plugin registrar,1,0.000
+stability checking,1,0.000
+describe_def,15,0.000
+variance testing,1,0.000
+codegen unit partitioning,1,0.000
+looking for entry point,1,0.000
+checking for inline asm in case the target doesn't support it,1,0.000
+inherent_impls,1,0.000
+crate_inherent_impls,1,0.000
+trait_of_item,7,0.000
+crate_inherent_impls_overlap_check,1,0.000
+attribute checking,1,0.000
+internalize symbols,1,0.000
+impl wf inference,1,0.000
+death checking,1,0.000
+reachability checking,1,0.000
+reachable_set,1,0.000
+is_exported_symbol,3,0.000
+is_mir_available,2,0.000
+unused lib feature checking,1,0.000
+maybe building test harness,1,0.000
+recursion limit,1,0.000
+write allocator module,1,0.000
+assert dep graph,1,0.000
+plugin registration,1,0.000
+write metadata,1,0.000
+
+
+We give some background about the query model of the Rust compiler.
+
+In the query model, many queries have a key that consists of a Def ID. The Rust
+compiler uses Def IDs to distinguish definitions in the input Rust program.
+From the compiler source code (src/librustc_middle/hir/def_id.rs):
+/// A DefId identifies a particular *definition*, by combining a crate
+/// index and a def index.
+#[derive(Clone, Eq, Ord, PartialOrd, PartialEq, RustcEncodable, RustcDecodable, Hash, Copy)]
+pub struct DefId {
+ pub krate: CrateNum,
+ pub index: DefIndex,
+}
+
+
+A query relates a key to a result, either by invoking a provider that
+computes this result, or by reusing a cached result that was provided earlier.
+We explain each term in more detail:
+
+- Query Provider: Each kind of query has a pre-defined provider, which
+refers to the compiler behavior that provides an answer to the query. These
+providers may nest; see trace of queries for more
+information about this nesting structure.
+Example providers:
+
+typeck_tables_of -- Typecheck a Def ID; produce "tables" of type
+information.borrowck -- Borrow-check a Def ID.optimized_mir -- Generate an optimized MIR for a Def ID; produce MIR.- For more examples, see Example 0.
+
+
+- Query Key: The input/arguments to the provider. Often, this consists of a
+particular Def ID.
+- Query Result: The output of the provider.
+
+
+Formally, a trace of the queries consists of a tree, where sub-trees
+represent sub-traces. In particular, the nesting structure of the trace of
+queries describes how the queries depend on one another.
+Even more precisely, this tree represents a directed acyclic graph (DAG), where
+shared sub-graphs consist of tree nodes that occur multiple times in the tree,
+first as "cache misses" and later as "cache hits".
+Cache hits and misses. The trace is a tree with the following possible tree
+nodes:
+
+- Query, with cache miss: The query's result is unknown, and its
+provider runs to compute it. In this case, the dynamic extent of the query's
+trace consists of the traced behavior of its provider.
+- Query, with cache hit: The query's result is known, and is reused; its
+provider does not rerun. These nodes are leaves in the trace, since they have
+no dynamic extent. These leaves also represent where the tree, represented as
+a DAG, would share a sub-graph (namely, the sub-graph of the query that was
+reused from the cache).
+
+Tree node metrics. To help determine how to style this tree, we define the
+following tree node metrics:
+
+- Depth: The number of ancestors of the node in its path from the tree
+root.
+- Extent: The number of immediate children of the node.
+
+Intuitively, a dependency tree is "good" for incremental caching when the depth
+and extent of each node is relatively small. It is pathological when either of
+these metrics grows too large. For instance, a tree node whose extent consists
+of 1M immediate children means that if and when this node is re-computed, all 1M
+children must be re-queried, at the very least (some may also require
+recomputation, too).
+
+Related design ideas, and tracking issues:
+
+More discussion and issues:
+
+
+This chapter is based on the explanation given by Niko Matsakis in this
+video about
+Salsa.
+
+Salsa is not used directly in rustc, but it is used extensively for
+rust-analyzer and may be integrated into the compiler in the future.
+
+
+Salsa is a library for incremental recomputation. This means it allows reusing
+computations that were already done in the past to increase the efficiency
+of future computations.
+The objectives of Salsa are:
+
+- Provide that functionality in an automatic way, so reusing old computations
+is done automatically by the library
+- Doing so in a "sound", or "correct", way, therefore leading to the same
+results as if it had been done from scratch
+
+Salsa's actual model is much richer, allowing many kinds of inputs and many
+different outputs.
+For example, integrating Salsa with an IDE could mean that the inputs could be
+the manifest (Cargo.toml), entire source files (foo.rs), snippets and so
+on; the outputs of such an integration could range from a binary executable, to
+lints, types (for example, if a user selects a certain variable and wishes to
+see its type), completions, etc.
+
+The first thing that Salsa has to do is identify the "base inputs" .
+Then Salsa has to also identify intermediate, "derived" values, which are
+something that the library produces, but, for each derived value there's a
+"pure" function that computes the derived value.
+For example, there might be a function ast(x: Path) -> AST. The produced
+AST isn't a final value, it's an intermidiate value that the library would
+use for the computation.
+This means that when you try to compute with the library, Salsa is going to
+compute various derived values, and eventually read the input and produce the
+result for the asked computation.
+In the course of computing, Salsa tracks which inputs were accessed and which
+values are derived. This information is used to determine what's going to
+happen when the inputs change: are the derived values still valid?
+This doesn't necessarily mean that each computation downstream from the input
+is going to be checked, which could be costly. Salsa only needs to check each
+downstream computation until it finds one that isn't changed. At that point, it
+won't check other derived computations since they wouldn't need to change.
+It's is helpful to think about this as a graph with nodes. Each derived value
+has a dependency on other values, which could themselves be either base or
+derived. Base values don't have a dependency.
+I <- A <- C ...
+ |
+J <- B <--+
+
+When an input I changes, the derived value A could change. The derived
+value B , which does not depend on I, A, or any value derived from A or
+I, is not subject to change. Therefore, Salsa can reuse the computation done
+for B in the past, without having to compute it again.
+The computation could also terminate early. Keeping the same graph as before,
+say that input I has changed in some way (and input J hasn't) but, when
+computing A again, it's found that A hasn't changed from the previous
+computation. This leads to an "early termination", because there's no need to
+check if C needs to change, since both C direct inputs, A and B,
+haven't changed.
+
+
+A query is some value that Salsa can access in the course of computation. Each
+query can have a number of keys (from 0 to many), and all queries have a
+result, akin to functions. 0-key queries are called "input" queries.
+
+The database is basically the context for the entire computation, it's meant to
+store Salsa's internal state, all intermediate values for each query, and
+anything else that the computation might need. The database must know all the
+queries that the library is going to do before it can be built, but they don't
+need to be specified in the same place.
+After the database is formed, it can be accessed with queries that are very
+similar to functions. Since each query's result is stored in the database,
+when a query is invoked N times, it will return N cloned results, without
+having to recompute the query (unless the input has changed in such a way that
+it warrants recomputation).
+For each input query (0-key), a "set" method is generated, allowing the user to
+change the output of such query, and trigger previous memoized values to be
+potentially invalidated.
+
+A query group is a set of queries which have been defined together as a unit.
+The database is formed by combining query groups. Query groups are akin to
+"Salsa modules" .
+A set of queries in a query group are just a set of methods in a trait.
+To create a query group a trait annotated with a specific attribute
+(#[salsa::query_group(...)]) has to be created.
+An argument must also be provided to said attribute as it will be used by Salsa
+to create a struct to be used later when the database is created.
+Example input query group:
+/// This attribute will process this tree, produce this tree as output, and produce
+/// a bunch of intermidiate stuff that Salsa also uses. One of these things is a
+/// "StorageStruct", whose name we have specified in the attribute.
+///
+/// This query group is a bunch of **input** queries, that do not rely on any
+/// derived input.
+#[salsa::query_group(InputsStorage)]
+pub trait Inputs {
+ /// This attribute (`#[salsa::input]`) indicates that this query is a base
+ /// input, therefore `set_manifest` is going to be auto-generated
+ #[salsa::input]
+ fn manifest(&self) -> Manifest;
+
+ #[salsa::input]
+ fn source_text(&self, name: String) -> String;
+}
+
+To create a derived query group, one must specify which other query groups
+this one depends on by specifying them as supertraits, as seen in the following
+example:
+/// This query group is going to contain queries that depend on derived values a
+/// query group can access another query group's queries by specifying the
+/// dependency as a super trait query groups can be stacked as much as needed using
+/// that pattern.
+#[salsa::query_group(ParserStorage)]
+pub trait Parser: Inputs {
+ /// This query `ast` is not an input query, it's a derived query this means
+ /// that a definition is necessary.
+ fn ast(&self, name: String) -> String;
+}
+
+When creating a derived query the implementation of said query must be defined
+outside the trait. The definition must take a database parameter as an impl Trait (or dyn Trait), where Trait is the query group that the definition
+belongs to, in addition to the other keys.
+///This is going to be the definition of the `ast` query in the `Parser` trait.
+///So, when the query `ast` is invoked, and it needs to be recomputed, Salsa is going to call this function
+///and it's is going to give it the database as `impl Parser`.
+///The function doesn't need to be aware of all the queries of all the query groups
+fn ast(db: &impl Parser, name: String) -> String {
+ //! Note, `impl Parser` is used here but `dyn Parser` works just as well
+ /* code */
+ ///By passing an `impl Parser`, this is allowed
+ let source_text = db.input_file(name);
+ /* do the actual parsing */
+ return ast;
+}
+
+Eventually, after all the query groups have been defined, the database can be
+created by declaring a struct.
+To specify which query groups are going to be part of the database an attribute
+(#[salsa::database(...)]) must be added. The argument of said attribute is a
+list of identifiers, specifying the query groups storages.
+///This attribute specifies which query groups are going to be in the database
+#[salsa::database(InputsStorage, ParserStorage)]
+#[derive(Default)] //optional!
+struct MyDatabase {
+ ///You also need this one field
+ runtime : salsa::Runtime<MyDatabase>,
+}
+///And this trait has to be implemented
+impl salsa::Databse for MyDatabase {
+ fn salsa_runtime(&self) -> &salsa::Runtime<MyDatabase> {
+ &self.runtime
+ }
+}
+
+Example usage:
+fn main() {
+ let db = MyDatabase::default();
+ db.set_manifest(...);
+ db.set_source_text(...);
+ loop {
+ db.ast(...); //will reuse results
+ db.set_source_text(...);
+ }
+}
+
+
+
+
+Rustc tries to be pretty careful how it manages memory. The compiler allocates
+a lot of data structures throughout compilation, and if we are not careful,
+it will take a lot of time and space to do so.
+One of the main way the compiler manages this is using arenas and interning.
+
+We create a LOT of data structures during compilation. For performance reasons,
+we allocate them from a global memory pool; they are each allocated once from a
+long-lived arena. This is called arena allocation. This system reduces
+allocations/deallocations of memory. It also allows for easy comparison of
+types for equality: for each interned type X, we implemented PartialEq for X, so we can just compare pointers. The CtxtInterners type
+contains a bunch of maps of interned types and the arena itself.
+
+Taking the example of ty::TyS which represents a type in the compiler (you
+can read more here). Each time we want to construct a type, the
+compiler doesn’t naively allocate from the buffer. Instead, we check if that
+type was already constructed. If it was, we just get the same pointer we had
+before, otherwise we make a fresh pointer. With this schema if we want to know
+if two types are the same, all we need to do is compare the pointers which is
+efficient. TyS is carefully setup so you never construct them on the stack.
+You always allocate them from this arena and you always intern them so they are
+unique.
+At the beginning of the compilation we make a buffer and each time we need to allocate a type we use
+some of this memory buffer. If we run out of space we get another one. The lifetime of that buffer
+is 'tcx. Our types are tied to that lifetime, so when compilation finishes all the memory related
+to that buffer is freed and our 'tcx references would be invalid.
+In addition to types, there are a number of other arena-allocated data structures that you can
+allocate, and which are found in this module. Here are a few examples:
+
+Substs, allocated with mk_substs – this will intern a slice of types, often used to
+specify the values to be substituted for generics (e.g. HashMap<i32, u32> would be represented
+as a slice &'tcx [tcx.types.i32, tcx.types.u32]).TraitRef, typically passed by value – a trait reference consists of a reference to a trait
+along with its various type parameters (including Self), like i32: Display (here, the def-id
+would reference the Display trait, and the substs would contain i32). Note that def-id is
+defined and discussed in depth in the AdtDef and DefId section.Predicate defines something the trait system has to prove (see traits module).
+
+The tcx ("typing context") is the central data structure in the compiler. It is the context that
+you use to perform all manner of queries. The struct TyCtxt defines a reference to this shared
+context:
+tcx: TyCtxt<'tcx>
+// ----
+// |
+// arena lifetime
+
+As you can see, the TyCtxt type takes a lifetime parameter. When you see a reference with a
+lifetime like 'tcx, you know that it refers to arena-allocated data (or data that lives as long as
+the arenas, anyhow).
+
+The Rust compiler is a fairly large program containing lots of big data
+structures (e.g. the AST, HIR, and the type system) and as such, arenas and
+references are heavily relied upon to minimize unnecessary memory use. This
+manifests itself in the way people can plug into the compiler (i.e. the
+driver), preferring a "push"-style API (callbacks) instead
+of the more Rust-ic "pull" style (think the Iterator trait).
+Thread-local storage and interning are used a lot through the compiler to reduce
+duplication while also preventing a lot of the ergonomic issues due to many
+pervasive lifetimes. The rustc::ty::tls module is used to access these
+thread-locals, although you should rarely need to touch it.
+
+Most of the compiler is not parallel. This represents an opportunity for
+improving compiler performance. Much effort has been put into parallelizing
+rustc, but it is still pretty early days for this work. There is a lot of
+design and correctness work that needs to be done.
+One can try out the current parallel compiler work by enabling it in the
+config.toml.
+There are a few basic ideas in this effort:
+
+- There are a lot of loops in the compiler that just iterate over all items in
+a crate. These can possibly be parallelized.
+- We can use (a custom fork of)
rayon to run tasks in parallel. The custom
+fork allows the execution of DAGs of tasks, not just trees.
+- There are currently a lot of global data structures that need to be made
+thread-safe. A key strategy here has been converting interior-mutable
+data-structures (e.g.
Cell) into their thread-safe siblings (e.g. Mutex).
+
+As of this writing, much of this effort is on hold due to lack of manpower. We
+have a working prototype with promising performance gains in many cases.
+However, there are two blockers:
+
+-
+
It's not clear what invariants need to be upheld that might not hold in the
+face of concurrency. An auditing effort was underway, but seems to have
+stalled at some point.
+
+-
+
There is a lot of lock contention, which actually degrades performance as the
+number of threads increases beyond 4.
+
+
+Here are some resources that can be used to learn more (note that some of them
+are a bit out of date):
+
+
+This part describes the process of taking raw source code from the user and
+transforming it into various forms that the compiler can work with easily.
+These are called intermediate representations (IRs).
+This process starts with compiler understanding what the user has asked for:
+parsing the command line arguments given and determining what it is to compile.
+After that, the compiler transforms the user input into a series of IRs that
+look progressively less like what the user wrote.
+
+Command-line flags are documented in the rustc book. All stable
+flags should be documented there. Unstable flags should be documented in the
+unstable book.
+
+
+- Flags should be orthogonal to each other. For example, if we'd have a
+json-emitting variant of multiple actions
foo and bar, an additional
+--json flag is better than adding --foo-json and --bar-json.
+- Avoid flags with the
no- prefix. Instead, use the parse_bool function,
+such as -C embed-bitcode=no.
+- Consider the behavior if the flag is passed multiple times. In some
+situations, the values should be accumulated (in order!). In other
+situations, subsequent flags should override previous flags (for example,
+the lint-level flags). And some flags (like
-o) should generate an error
+if it is too ambiguous what multiple flags would mean.
+- Always give options a long descriptive name, if only for more understandable
+compiler scripts.
+- The
--verbose flag is for adding verbose information to rustc output
+when not compiling a program. For example, using it with the --version
+flag gives information about the hashes of the compiler code.
+- Experimental flags and options must be guarded behind the
-Z unstable-options flag.
+
+
+The rustc_driver is essentially rustc's main() function. It acts as
+the glue for running the various phases of the compiler in the correct order,
+using the interface defined in the rustc_interface crate.
+The rustc_interface crate provides external users with an (unstable) API
+for running code at particular times during the compilation process, allowing
+third parties to effectively use rustc's internals as a library for
+analysing a crate or emulating the compiler in-process (e.g. the RLS or rustdoc).
+For those using rustc as a library, the rustc_interface::run_compiler()
+function is the main entrypoint to the compiler. It takes a configuration for the compiler
+and a closure that takes a Compiler. run_compiler creates a Compiler from the
+configuration and passes it to the closure. Inside the closure, you can use the Compiler
+to drive queries to compile a crate and get the results. This is what the rustc_driver does too.
+You can see a minimal example of how to use rustc_interface here.
+You can see what queries are currently available through the rustdocs for Compiler.
+You can see an example of how to use them by looking at the rustc_driver implementation,
+specifically the rustc_driver::run_compiler function (not to be confused with
+rustc_interface::run_compiler). The rustc_driver::run_compiler function
+takes a bunch of command-line args and some other configurations and
+drives the compilation to completion.
+rustc_driver::run_compiler also takes a Callbacks,
+a trait that allows for custom compiler configuration,
+as well as allowing some custom code run after different phases of the compilation.
+
+Warning: By its very nature, the internal compiler APIs are always going
+to be unstable. That said, we do try not to break things unnecessarily.
+
+
+Rustdoc actually uses the rustc internals directly. It lives in-tree with the
+compiler and standard library. This chapter is about how it works.
+Rustdoc is implemented entirely within the crate librustdoc. It runs
+the compiler up to the point where we have an internal representation of a
+crate (HIR) and the ability to run some queries about the types of items. HIR
+and queries are discussed in the linked chapters.
+librustdoc performs two major steps after that to render a set of
+documentation:
+
+- "Clean" the AST into a form that's more suited to creating documentation (and
+slightly more resistant to churn in the compiler).
+- Use this cleaned AST to render a crate's documentation, one page at a time.
+
+Naturally, there's more than just this, and those descriptions simplify out
+lots of details, but that's the high-level overview.
+(Side note: librustdoc is a library crate! The rustdoc binary is created
+using the project in src/tools/rustdoc. Note that literally all that
+does is call the main() that's in this crate's lib.rs, though.)
+
+
+- Use
./x.py build --stage 1 src/libstd src/tools/rustdoc to make a usable
+rustdoc you can run on other projects.
+
+- Add
src/libtest to be able to use rustdoc --test.
+- If you've used
rustup toolchain link local /path/to/build/$TARGET/stage1
+previously, then after the previous build command, cargo +local doc will
+Just Work.
+
+
+- Use
./x.py doc --stage 1 src/libstd to use this rustdoc to generate the
+standard library docs.
+
+- The completed docs will be available in
build/$TARGET/doc/std, though the
+bundle is meant to be used as though you would copy out the doc folder to
+a web server, since that's where the CSS/JS and landing page are.
+
+
+- Most of the HTML printing code is in
html/format.rs and html/render.rs.
+It's in a bunch of fmt::Display implementations and supplementary
+functions.
+- The types that got
Display impls above are defined in clean/mod.rs, right
+next to the custom Clean trait used to process them out of the rustc HIR.
+- The bits specific to using rustdoc as a test harness are in
test.rs.
+- The Markdown renderer is loaded up in
html/markdown.rs, including functions
+for extracting doctests from a given block of Markdown.
+- The tests on rustdoc output are located in
src/test/rustdoc, where
+they're handled by the test runner of rustbuild and the supplementary script
+src/etc/htmldocck.py.
+- Tests on search index generation are located in
src/test/rustdoc-js, as a
+series of JavaScript files that encode queries on the standard library search
+index and expected results.
+
+
+In core.rs are two central items: the DocContext struct, and the run_core
+function. The latter is where rustdoc calls out to rustc to compile a crate to
+the point where rustdoc can take over. The former is a state container used
+when crawling through a crate to gather its documentation.
+The main process of crate crawling is done in clean/mod.rs through several
+implementations of the Clean trait defined within. This is a conversion
+trait, which defines one method:
+pub trait Clean<T> {
+ fn clean(&self, cx: &DocContext) -> T;
+}
+
+clean/mod.rs also defines the types for the "cleaned" AST used later on to
+render documentation pages. Each usually accompanies an implementation of
+Clean that takes some AST or HIR type from rustc and converts it into the
+appropriate "cleaned" type. "Big" items like modules or associated items may
+have some extra processing in its Clean implementation, but for the most part
+these impls are straightforward conversions. The "entry point" to this module
+is the impl Clean<Crate> for visit_ast::RustdocVisitor, which is called by
+run_core above.
+You see, I actually lied a little earlier: There's another AST transformation
+that happens before the events in clean/mod.rs. In visit_ast.rs is the
+type RustdocVisitor, which actually crawls a rustc_hir::Crate to get the first
+intermediate representation, defined in doctree.rs. This pass is mainly to
+get a few intermediate wrappers around the HIR types and to process visibility
+and inlining. This is where #[doc(inline)], #[doc(no_inline)], and
+#[doc(hidden)] are processed, as well as the logic for whether a pub use
+should get the full page or a "Reexport" line in the module page.
+The other major thing that happens in clean/mod.rs is the collection of doc
+comments and #[doc=""] attributes into a separate field of the Attributes
+struct, present on anything that gets hand-written documentation. This makes it
+easier to collect this documentation later in the process.
+The primary output of this process is a clean::Crate with a tree of Items
+which describe the publicly-documentable items in the target crate.
+
+Before moving on to the next major step, a few important "passes" occur over
+the documentation. These do things like combine the separate "attributes" into
+a single string and strip leading whitespace to make the document easier on the
+markdown parser, or drop items that are not public or deliberately hidden with
+#[doc(hidden)]. These are all implemented in the passes/ directory, one
+file per pass. By default, all of these passes are run on a crate, but the ones
+regarding dropping private/hidden items can be bypassed by passing
+--document-private-items to rustdoc. Note that unlike the previous set of AST
+transformations, the passes happen on the cleaned crate.
+(Strictly speaking, you can fine-tune the passes run and even add your own, but
+we're trying to deprecate that. If you need finer-grain control over
+these passes, please let us know!)
+Here is current (as of this writing) list of passes:
+
+propagate-doc-cfg - propagates #[doc(cfg(...))] to child items.collapse-docs concatenates all document attributes into one document
+attribute. This is necessary because each line of a doc comment is given as a
+separate doc attribute, and this will combine them into a single string with
+line breaks between each attribute.unindent-comments removes excess indentation on comments in order for
+markdown to like it. This is necessary because the convention for writing
+documentation is to provide a space between the /// or //! marker and the
+text, and stripping that leading space will make the text easier to parse by
+the Markdown parser. (In the past, the markdown parser used was not
+Commonmark- compliant, which caused annoyances with extra whitespace but this
+seems to be less of an issue today.)strip-priv-imports strips all private import statements (use, extern crate) from a crate. This is necessary because rustdoc will handle public
+imports by either inlining the item's documentation to the module or creating
+a "Reexports" section with the import in it. The pass ensures that all of
+these imports are actually relevant to documentation.strip-hidden and strip-private strip all doc(hidden) and private items
+from the output. strip-private implies strip-priv-imports. Basically, the
+goal is to remove items that are not relevant for public documentation.
+
+This is where the "second phase" in rustdoc begins. This phase primarily lives
+in the html/ folder, and it all starts with run() in html/render.rs. This
+code is responsible for setting up the Context, SharedContext, and Cache
+which are used during rendering, copying out the static files which live in
+every rendered set of documentation (things like the fonts, CSS, and JavaScript
+that live in html/static/), creating the search index, and printing out the
+source code rendering, before beginning the process of rendering all the
+documentation for the crate.
+Several functions implemented directly on Context take the clean::Crate and
+set up some state between rendering items or recursing on a module's child
+items. From here the "page rendering" begins, via an enormous write!() call
+in html/layout.rs. The parts that actually generate HTML from the items and
+documentation occurs within a series of std::fmt::Display implementations and
+functions that pass around a &mut std::fmt::Formatter. The top-level
+implementation that writes out the page body is the impl<'a> fmt::Display for Item<'a> in html/render.rs, which switches out to one of several item_*
+functions based on the kind of Item being rendered.
+Depending on what kind of rendering code you're looking for, you'll probably
+find it either in html/render.rs for major items like "what sections should I
+print for a struct page" or html/format.rs for smaller component pieces like
+"how should I print a where clause as part of some other item".
+Whenever rustdoc comes across an item that should print hand-written
+documentation alongside, it calls out to html/markdown.rs which interfaces
+with the Markdown parser. This is exposed as a series of types that wrap a
+string of Markdown, and implement fmt::Display to emit HTML text. It takes
+special care to enable certain features like footnotes and tables and add
+syntax highlighting to Rust code blocks (via html/highlight.rs) before
+running the Markdown parser. There's also a function in here
+(find_testable_code) that specifically scans for Rust code blocks so the
+test-runner code can find all the doctests in the crate.
+
+(alternate title: "An unbroken thread that stretches from those first Cells
+to us")
+It's important to note that the AST cleaning can ask the compiler for
+information (crucially, DocContext contains a TyCtxt), but page rendering
+cannot. The clean::Crate created within run_core is passed outside the
+compiler context before being handed to html::render::run. This means that a
+lot of the "supplementary data" that isn't immediately available inside an
+item's definition, like which trait is the Deref trait used by the language,
+needs to be collected during cleaning, stored in the DocContext, and passed
+along to the SharedContext during HTML rendering. This manifests as a bunch
+of shared state, context variables, and RefCells.
+Also of note is that some items that come from "asking the compiler" don't go
+directly into the DocContext - for example, when loading items from a foreign
+crate, rustdoc will ask about trait implementations and generate new Items
+for the impls based on that information. This goes directly into the returned
+Crate rather than roundabout through the DocContext. This way, these
+implementations can be collected alongside the others, right before rendering
+the HTML.
+
+All this describes the process for generating HTML documentation from a Rust
+crate, but there are couple other major modes that rustdoc runs in. It can also
+be run on a standalone Markdown file, or it can run doctests on Rust code or
+standalone Markdown files. For the former, it shortcuts straight to
+html/markdown.rs, optionally including a mode which inserts a Table of
+Contents to the output HTML.
+For the latter, rustdoc runs a similar partial-compilation to get relevant
+documentation in test.rs, but instead of going through the full clean and
+render process, it runs a much simpler crate walk to grab just the
+hand-written documentation. Combined with the aforementioned
+"find_testable_code" in html/markdown.rs, it builds up a collection of
+tests to run before handing them off to the libtest test runner. One notable
+location in test.rs is the function make_test, which is where hand-written
+doctests get transformed into something that can be executed.
+Some extra reading about make_test can be found
+here.
+
+So that's rustdoc's code in a nutshell, but there's more things in the repo
+that deal with it. Since we have the full compiletest suite at hand, there's
+a set of tests in src/test/rustdoc that make sure the final HTML is what we
+expect in various situations. These tests also use a supplementary script,
+src/etc/htmldocck.py, that allows it to look through the final HTML using
+XPath notation to get a precise look at the output. The full description of all
+the commands available to rustdoc tests is in htmldocck.py.
+In addition, there are separate tests for the search index and rustdoc's
+ability to query it. The files in src/test/rustdoc-js each contain a
+different search query and the expected results, broken out by search tab.
+These files are processed by a script in src/tools/rustdoc-js and the Node.js
+runtime. These tests don't have as thorough of a writeup, but a broad example
+that features results in all tabs can be found in basic.js. The basic idea is
+that you match a given QUERY with a set of EXPECTED results, complete with
+the full item path of each item.
+
+rustc_interface allows you to interact with Rust code at various stages of compilation.
+
+To get the type of an expression, use the global_ctxt to get a TyCtxt:
+// See https://github.com/rust-lang/rustc-dev-guide/blob/master/examples/rustc-driver-interacting-with-the-ast.rs for complete program.
+let config = rustc_interface::Config {
+ input: config::Input::Str {
+ name: source_map::FileName::Custom("main.rs".to_string()),
+ input: "fn main() { let message = \"Hello, world!\"; println!(\"{}\", message); }"
+ .to_string(),
+ },
+ /* other config */
+};
+rustc_interface::run_compiler(config, |compiler| {
+ compiler.enter(|queries| {
+ // Analyze the crate and inspect the types under the cursor.
+ queries.global_ctxt().unwrap().take().enter(|tcx| {
+ // Every compilation contains a single crate.
+ let krate = tcx.hir().krate();
+ // Iterate over the top-level items in the crate, looking for the main function.
+ for (_, item) in &krate.items {
+ // Use pattern-matching to find a specific node inside the main function.
+ if let rustc_hir::ItemKind::Fn(_, _, body_id) = item.kind {
+ let expr = &tcx.hir().body(body_id).value;
+ if let rustc_hir::ExprKind::Block(block, _) = expr.kind {
+ if let rustc_hir::StmtKind::Local(local) = block.stmts[0].kind {
+ if let Some(expr) = local.init {
+ let hir_id = expr.hir_id; // hir_id identifies the string "Hello, world!"
+ let def_id = tcx.hir().local_def_id(item.hir_id); // def_id identifies the main function
+ let ty = tcx.typeck_tables_of(def_id).node_type(hir_id);
+ println!("{:?}: {:?}", expr, ty); // prints expr(HirId { owner: DefIndex(3), local_id: 4 }: "Hello, world!"): &'static str
+ }
+ }
+ }
+ }
+ }
+ })
+ });
+});
+
+
+rustc_interface allows you to intercept diagnostics that would otherwise be printed to stderr.
+
+To get diagnostics from the compiler,
+configure rustc_interface::Config to output diagnostic to a buffer,
+and run TyCtxt.analysis:
+
+#![allow(unused_variables)]
+fn main() {
+// See https://github.com/rust-lang/rustc-dev-guide/blob/master/examples/rustc-driver-getting-diagnostics.rs for complete program.
+let buffer = sync::Arc::new(sync::Mutex::new(Vec::new()));
+let config = rustc_interface::Config {
+ opts: config::Options {
+ // Configure the compiler to emit diagnostics in compact JSON format.
+ error_format: config::ErrorOutputType::Json {
+ pretty: false,
+ json_rendered: rustc_errors::emitter::HumanReadableErrorType::Default(
+ rustc_errors::emitter::ColorConfig::Never,
+ ),
+ },
+ /* other config */
+ },
+ // Redirect the diagnostic output of the compiler to a buffer.
+ diagnostic_output: rustc_session::DiagnosticOutput::Raw(Box::from(DiagnosticSink(
+ buffer.clone(),
+ ))),
+ /* other config */
+};
+rustc_interface::run_compiler(config, |compiler| {
+ compiler.enter(|queries| {
+ queries.global_ctxt().unwrap().take().enter(|tcx| {
+ // Run the analysis phase on the local crate to trigger the type error.
+ tcx.analysis(rustc_hir::def_id::LOCAL_CRATE);
+ });
+ });
+});
+// Read buffered diagnostics.
+let diagnostics = String::from_utf8(buffer.lock().unwrap().clone()).unwrap();
+}
+
+
+Working directly with source code is very inconvenient and error-prone. Thus,
+before we do anything else, we convert raw source code into an AST. It turns
+out that doing even this involves a lot of work, including lexing, parsing,
+macro expansion, name resolution, conditional compilation, feature-gate
+checking, and validation of the AST. In this chapter, we take a look at all
+of these steps.
+Notably, there isn't always a clean ordering between these tasks. For example,
+macro expansion relies on name resolution to resolve the names of macros and
+imports. And parsing requires macro expansion, which in turn may require
+parsing the output of the macro.
+
+
+The parser and lexer are currently undergoing a lot of refactoring, so parts
+of this chapter may be out of date.
+
+The very first thing the compiler does is take the program (in Unicode
+characters) and turn it into something the compiler can work with more
+conveniently than strings. This happens in two stages: Lexing and Parsing.
+Lexing takes strings and turns them into streams of tokens. For example,
+a.b + c would be turned into the tokens a, ., b, +, and c.
+The lexer lives in librustc_lexer.
+Parsing then takes streams of tokens and turns them into a structured
+form which is easier for the compiler to work with, usually called an Abstract
+Syntax Tree (AST). An AST mirrors the structure of a Rust program in memory,
+using a Span to link a particular AST node back to its source text.
+The AST is defined in librustc_ast, along with some definitions for
+tokens and token streams, data structures/traits for mutating ASTs, and shared
+definitions for other AST-related parts of the compiler (like the lexer and
+macro-expansion).
+The parser is defined in librustc_parse, along with a
+high-level interface to the lexer and some validation routines that run after
+macro expansion. In particular, the rustc_parse::parser contains
+the parser implementation.
+The main entrypoint to the parser is via the various parse_* functions in the
+parser. They let you do things like turn a SourceFile
+(e.g. the source in a single file) into a token stream, create a parser from
+the token stream, and then execute the parser to get a Crate (the root AST
+node).
+To minimise the amount of copying that is done, both the StringReader and
+Parser have lifetimes which bind them to the parent ParseSess. This contains
+all the information needed while parsing, as well as the SourceMap itself.
+Note that while parsing, we may encounter macro definitions or invocations. We
+set these aside to be expanded (see this chapter).
+Expansion may itself require parsing the output of the macro, which may reveal
+more macros to be expanded, and so on.
+
+Code for lexical analysis is split between two crates:
+
+-
+
rustc_lexer crate is responsible for breaking a &str into chunks
+constituting tokens. Although it is popular to implement lexers as generated
+finite state machines, the lexer in rustc_lexer is hand-written.
+
+-
+
StringReader from librustc_ast integrates rustc_lexer with rustc
+specific data structures. Specifically, it adds Span information to tokens
+returned by rustc_lexer and interns identifiers.
+
+
+
+
+librustc_ast, librustc_expand, and librustc_builtin_macros are all undergoing
+refactoring, so some of the links in this chapter may be broken.
+
+Rust has a very powerful macro system. In the previous chapter, we saw how the
+parser sets aside macros to be expanded (it temporarily uses placeholders).
+This chapter is about the process of expanding those macros iteratively until
+we have a complete AST for our crate with no unexpanded macros (or a compile
+error).
+First, we will discuss the algorithm that expands and integrates macro output
+into ASTs. Next, we will take a look at how hygiene data is collected. Finally,
+we will look at the specifics of expanding different types of macros.
+Many of the algorithms and data structures described below are in rustc_expand,
+with basic data structures in rustc_expand::base.
+Also of note, cfg and cfg_attr are treated specially from other macros, and are
+handled in rustc_expand::config.
+
+First of all, expansion happens at the crate level. Given a raw source code for
+a crate, the compiler will produce a massive AST with all macros expanded, all
+modules inlined, etc. The primary entry point for this process is the
+MacroExpander::fully_expand_fragment method. With few exceptions, we
+use this method on the whole crate (see "Eager Expansion"
+below for more detailed discussion of edge case expansion issues).
+At a high level, fully_expand_fragment works in iterations. We keep a
+queue of unresolved macro invocations (that is, macros we haven't found the
+definition of yet). We repeatedly try to pick a macro from the queue, resolve
+it, expand it, and integrate it back. If we can't make progress in an
+iteration, this represents a compile error. Here is the algorithm:
+
+- Initialize an
queue of unresolved macros.
+- Repeat until
queue is empty (or we make no progress, which is an error):
+
+- Resolve imports in our partially built crate as
+much as possible.
+- Collect as many macro
Invocations as possible from our
+partially built crate (fn-like, attributes, derives) and add them to the
+queue.
+- Dequeue the first element, and attempt to resolve it.
+- If it's resolved:
+
+- Run the macro's expander function that consumes a
TokenStream or
+AST and produces a TokenStream or AstFragment (depending on
+the macro kind). (A TokenStream is a collection of TokenTrees,
+each of which are a token (punctuation, identifier, or literal) or a
+delimited group (anything inside ()/[]/{})).
+
+- At this point, we know everything about the macro itself and can
+call
set_expn_data to fill in its properties in the global data;
+that is the hygiene data associated with ExpnId. (See the
+"Hygiene" section below).
+
+
+- Integrate that piece of AST into the big existing partially built
+AST. This is essentially where the "token-like mass" becomes a
+proper set-in-stone AST with side-tables. It happens as follows:
+
+- If the macro produces tokens (e.g. a proc macro), we parse into
+an AST, which may produce parse errors.
+- During expansion, we create
SyntaxContexts (hierarchy 2). (See
+the "Hygiene" section below)
+- These three passes happen one after another on every AST fragment
+freshly expanded from a macro:
+
+
+
+
+- After expanding a single macro and integrating its output, continue
+to the next iteration of
fully_expand_fragment.
+
+
+- If it's not resolved:
+
+- Put the macro back in the queue
+- Continue to next iteration...
+
+
+
+
+
+
+If we make no progress in an iteration, then we have reached a compilation
+error (e.g. an undefined macro). We attempt to recover from failures
+(unresolved macros or imports) for the sake of diagnostics. This allows
+compilation to continue past the first error, so that we can report more errors
+at a time. Recovery can't cause compilation to suceed. We know that it will
+fail at this point. The recovery happens by expanding unresolved macros into
+ExprKind::Err.
+
+Notice that name resolution is involved here: we need to resolve imports and
+macro names in the above algorithm. This is done in
+rustc_resolve::macros, which resolves macro paths, validates
+those resolutions, and reports various errors (e.g. "not found" or "found, but
+it's unstable" or "expected x, found y"). However, we don't try to resolve
+other names yet. This happens later, as we will see in the next
+chapter.
+
+Eager expansion means that we expand the arguments of a macro invocation
+before the macro invocation itself. This is implemented only for a few special
+built-in macros that expect literals; expanding arguments first for some of
+these macro results in a smoother user experience. As an example, consider the
+following:
+macro bar($i: ident) { $i }
+macro foo($i: ident) { $i }
+
+foo!(bar!(baz));
+
+A lazy expansion would expand foo! first. An eager expansion would expand
+bar! first.
+Eager expansion is not a generally available feature of Rust. Implementing
+eager expansion more generally would be challenging, but we implement it for a
+few special built-in macros for the sake of user experience. The built-in
+macros are implemented in rustc_builtin_macros, along with some other early
+code generation facilities like injection of standard library imports or
+generation of test harness. There are some additional helpers for building
+their AST fragments in rustc_expand::build. Eager expansion generally
+performs a subset of the things that lazy (normal) expansion. It is done by
+invoking fully_expand_fragment on only part of a crate (as opposed to
+whole crate, like we normally do).
+
+Here are some other notable data structures involved in expansion and integration:
+
+Resolver - a trait used to break crate dependencies. This allows the
+resolver services to be used in rustc_ast, despite rustc_resolve and
+pretty much everything else depending on rustc_ast.ExtCtxt/ExpansionData - various intermediate data kept and used by expansion
+infrastructure in the process of its workAnnotatable - a piece of AST that can be an attribute target, almost same
+thing as AstFragment except for types and patterns that can be produced by
+macros but cannot be annotated with attributesMacResult - a "polymorphic" AST fragment, something that can turn into a
+different AstFragment depending on its AstFragmentKind - item,
+or expression, or pattern etc.
+
+If you have ever used C/C++ preprocessor macros, you know that there are some
+annoying and hard-to-debug gotchas! For example, consider the following C code:
+#define DEFINE_FOO struct Bar {int x;}; struct Foo {Bar bar;};
+
+// Then, somewhere else
+struct Bar {
+ ...
+};
+
+DEFINE_FOO
+
+Most people avoid writing C like this – and for good reason: it doesn't
+compile. The struct Bar defined by the macro clashes names with the struct Bar defined in the code. Consider also the following example:
+#define DO_FOO(x) {\
+ int y = 0;\
+ foo(x, y);\
+ }
+
+// Then elsewhere
+int y = 22;
+DO_FOO(y);
+
+Do you see the problem? We wanted to generate a call foo(22, 0), but instead
+we got foo(0, 0) because the macro defined its own y!
+These are both examples of macro hygiene issues. Hygiene relates to how to
+handle names defined within a macro. In particular, a hygienic macro system
+prevents errors due to names introduced within a macro. Rust macros are hygienic
+in that they do not allow one to write the sorts of bugs above.
+At a high level, hygiene within the rust compiler is accomplished by keeping
+track of the context where a name is introduced and used. We can then
+disambiguate names based on that context. Future iterations of the macro system
+will allow greater control to the macro author to use that context. For example,
+a macro author may want to introduce a new name to the context where the macro
+was called. Alternately, the macro author may be defining a variable for use
+only within the macro (i.e. it should not be visible outside the macro).
+The context is attached to AST nodes. All AST nodes generated by macros have
+context attached. Additionally, there may be other nodes that have context
+attached, such as some desugared syntax (non-macro-expanded nodes are
+considered to just have the "root" context, as described below).
+Throughout the compiler, we use librustc_span::Spans to refer to code locations.
+This struct also has hygiene information attached to it, as we will see later.
+Because macros invocations and definitions can be nested, the syntax context of
+a node must be a hierarchy. For example, if we expand a macro and there is
+another macro invocation or definition in the generated output, then the syntax
+context should reflex the nesting.
+However, it turns out that there are actually a few types of context we may
+want to track for different purposes. Thus, there are not just one but three
+expansion hierarchies that together comprise the hygiene information for a
+crate.
+All of these hierarchies need some sort of "macro ID" to identify individual
+elements in the chain of expansions. This ID is ExpnId. All macros receive
+an integer ID, assigned continuously starting from 0 as we discover new macro
+calls. All hierarchies start at ExpnId::root(), which is its own
+parent.
+rustc_span::hygiene contains all of the hygiene-related algorithms
+(with the exception of some hacks in Resolver::resolve_crate_root)
+and structures related to hygiene and expansion that are kept in global data.
+The actual hierarchies are stored in HygieneData. This is a global
+piece of data containing hygiene and expansion info that can be accessed from
+any Ident without any context.
+
+The first hierarchy tracks the order of expansions, i.e., when a macro
+invocation is in the output of another macro.
+Here, the children in the hierarchy will be the "innermost" tokens. The
+ExpnData struct itself contains a subset of properties from both macro
+definition and macro call available through global data.
+ExpnData::parent tracks the child -> parent link in this hierarchy.
+For example,
+macro_rules! foo { () => { println!(); } }
+
+fn main() { foo!(); }
+
+In this code, the AST nodes that are finally generated would have hierarchy:
+root
+ expn_id_foo
+ expn_id_println
+
+
+The second hierarchy tracks the order of macro definitions, i.e., when we are
+expanding one macro another macro definition is revealed in its output. This
+one is a bit tricky and more complex than the other two hierarchies.
+SyntaxContext represents a whole chain in this hierarchy via an ID.
+SyntaxContextData contains data associated with the given
+SyntaxContext; mostly it is a cache for results of filtering that chain in
+different ways. SyntaxContextData::parent is the child -> parent
+link here, and SyntaxContextData::outer_expns are individual
+elements in the chain. The "chaining operator" is
+SyntaxContext::apply_mark in compiler code.
+A Span, mentioned above, is actually just a compact representation of
+a code location and SyntaxContext. Likewise, an Ident is just an interned
+Symbol + Span (i.e. an interned string + hygiene data).
+For built-in macros, we use the context:
+SyntaxContext::empty().apply_mark(expn_id), and such macros are considered to
+be defined at the hierarchy root. We do the same for proc-macros because we
+haven't implemented cross-crate hygiene yet.
+If the token had context X before being produced by a macro then after being
+produced by the macro it has context X -> macro_id. Here are some examples:
+Example 0:
+macro m() { ident }
+
+m!();
+
+Here ident originally has context SyntaxContext::root(). ident has
+context ROOT -> id(m) after it's produced by m.
+Example 1:
+macro m() { macro n() { ident } }
+
+m!();
+n!();
+
+In this example the ident has context ROOT originally, then ROOT -> id(m)
+after the first expansion, then ROOT -> id(m) -> id(n).
+Example 2:
+Note that these chains are not entirely determined by their last element, in
+other words ExpnId is not isomorphic to SyntaxContext.
+macro m($i: ident) { macro n() { ($i, bar) } }
+
+m!(foo);
+
+After all expansions, foo has context ROOT -> id(n) and bar has context
+ROOT -> id(m) -> id(n).
+Finally, one last thing to mention is that currently, this hierarchy is subject
+to the "context transplantation hack". Basically, the more modern (and
+experimental) macro macros have stronger hygiene than the older MBE system,
+but this can result in weird interactions between the two. The hack is intended
+to make things "just work" for now.
+
+The third and final hierarchy tracks the location of macro invocations.
+In this hierarchy ExpnData::call_site is the child -> parent link.
+Here is an example:
+macro bar($i: ident) { $i }
+macro foo($i: ident) { $i }
+
+foo!(bar!(baz));
+
+For the baz AST node in the final output, the first hierarchy is ROOT -> id(foo) -> id(bar) -> baz, while the third hierarchy is ROOT -> baz.
+
+Macro backtraces are implemented in rustc_span using the hygiene machinery
+in rustc_span::hygiene.
+
+Above, we saw how the output of a macro is integrated into the AST for a crate,
+and we also saw how the hygiene data for a crate is generated. But how do we
+actually produce the output of a macro? It depends on the type of macro.
+There are two types of macros in Rust:
+macro_rules! macros (a.k.a. "Macros By Example" (MBE)) and procedural macros
+(or "proc macros"; including custom derives). During the parsing phase, the normal
+Rust parser will set aside the contents of macros and their invocations. Later,
+macros are expanded using these portions of the code.
+Some important data structures/interfaces here:
+
+SyntaxExtension - a lowered macro representation, contains its expander
+function, which transforms a TokenStream or AST into another TokenStream
+or AST + some additional data like stability, or a list of unstable features
+allowed inside the macro.SyntaxExtensionKind - expander functions may have several different
+signatures (take one token stream, or two, or a piece of AST, etc). This is
+an enum that lists them.ProcMacro/TTMacroExpander/AttrProcMacro/MultiItemModifier -
+traits representing the expander function signatures.
+
+MBEs have their own parser distinct from the normal Rust parser. When macros
+are expanded, we may invoke the MBE parser to parse and expand a macro. The
+MBE parser, in turn, may call the normal Rust parser when it needs to bind a
+metavariable (e.g. $my_expr) while parsing the contents of a macro
+invocation. The code for macro expansion is in
+src/librustc_expand/mbe/.
+
+It's helpful to have an example to refer to. For the remainder of this chapter,
+whenever we refer to the "example definition", we mean the following:
+macro_rules! printer {
+ (print $mvar:ident) => {
+ println!("{}", $mvar);
+ };
+ (print twice $mvar:ident) => {
+ println!("{}", $mvar);
+ println!("{}", $mvar);
+ };
+}
+
+$mvar is called a metavariable. Unlike normal variables, rather than
+binding to a value in a computation, a metavariable binds at compile time to
+a tree of tokens. A token is a single "unit" of the grammar, such as an
+identifier (e.g. foo) or punctuation (e.g. =>). There are also other
+special tokens, such as EOF, which indicates that there are no more tokens.
+Token trees resulting from paired parentheses-like characters ((...),
+[...], and {...}) – they include the open and close and all the tokens
+in between (we do require that parentheses-like characters be balanced). Having
+macro expansion operate on token streams rather than the raw bytes of a source
+file abstracts away a lot of complexity. The macro expander (and much of the
+rest of the compiler) doesn't really care that much about the exact line and
+column of some syntactic construct in the code; it cares about what constructs
+are used in the code. Using tokens allows us to care about what without
+worrying about where. For more information about tokens, see the
+Parsing chapter of this book.
+Whenever we refer to the "example invocation", we mean the following snippet:
+printer!(print foo); // Assume `foo` is a variable defined somewhere else...
+
+The process of expanding the macro invocation into the syntax tree
+println!("{}", foo) and then expanding that into a call to Display::fmt is
+called macro expansion, and it is the topic of this chapter.
+
+There are two parts to MBE expansion: parsing the definition and parsing the
+invocations. Interestingly, both are done by the macro parser.
+Basically, the MBE parser is like an NFA-based regex parser. It uses an
+algorithm similar in spirit to the Earley parsing
+algorithm. The macro parser is
+defined in src/librustc_expand/mbe/macro_parser.rs.
+The interface of the macro parser is as follows (this is slightly simplified):
+fn parse_tt(
+ parser: &mut Cow<Parser>,
+ ms: &[TokenTree],
+) -> NamedParseResult
+
+We use these items in macro parser:
+
+parser is a reference to the state of a normal Rust parser, including the
+token stream and parsing session. The token stream is what we are about to
+ask the MBE parser to parse. We will consume the raw stream of tokens and
+output a binding of metavariables to corresponding token trees. The parsing
+session can be used to report parser errros.ms a matcher. This is a sequence of token trees that we want to match
+the token stream against.
+In the analogy of a regex parser, the token stream is the input and we are matching it
+against the pattern ms. Using our examples, the token stream could be the stream of
+tokens containing the inside of the example invocation print foo, while ms
+might be the sequence of token (trees) print $mvar:ident.
+The output of the parser is a NamedParseResult, which indicates which of
+three cases has occurred:
+
+- Success: the token stream matches the given matcher
ms, and we have produced a binding
+from metavariables to the corresponding token trees.
+- Failure: the token stream does not match
ms. This results in an error message such as
+"No rule expected token blah".
+- Error: some fatal error has occurred in the parser. For example, this
+happens if there are more than one pattern match, since that indicates
+the macro is ambiguous.
+
+The full interface is defined here.
+The macro parser does pretty much exactly the same as a normal regex parser with
+one exception: in order to parse different types of metavariables, such as
+ident, block, expr, etc., the macro parser must sometimes call back to the
+normal Rust parser.
+As mentioned above, both definitions and invocations of macros are parsed using
+the macro parser. This is extremely non-intuitive and self-referential. The code
+to parse macro definitions is in
+src/librustc_expand/mbe/macro_rules.rs. It defines the pattern for
+matching for a macro definition as $( $lhs:tt => $rhs:tt );+. In other words,
+a macro_rules definition should have in its body at least one occurrence of a
+token tree followed by => followed by another token tree. When the compiler
+comes to a macro_rules definition, it uses this pattern to match the two token
+trees per rule in the definition of the macro using the macro parser itself.
+In our example definition, the metavariable $lhs would match the patterns of
+both arms: (print $mvar:ident) and (print twice $mvar:ident). And $rhs
+would match the bodies of both arms: { println!("{}", $mvar); } and { println!("{}", $mvar); println!("{}", $mvar); }. The parser would keep this
+knowledge around for when it needs to expand a macro invocation.
+When the compiler comes to a macro invocation, it parses that invocation using
+the same NFA-based macro parser that is described above. However, the matcher
+used is the first token tree ($lhs) extracted from the arms of the macro
+definition. Using our example, we would try to match the token stream print foo from the invocation against the matchers print $mvar:ident and print twice $mvar:ident that we previously extracted from the definition. The
+algorithm is exactly the same, but when the macro parser comes to a place in the
+current matcher where it needs to match a non-terminal (e.g. $mvar:ident),
+it calls back to the normal Rust parser to get the contents of that
+non-terminal. In this case, the Rust parser would look for an ident token,
+which it finds (foo) and returns to the macro parser. Then, the macro parser
+proceeds in parsing as normal. Also, note that exactly one of the matchers from
+the various arms should match the invocation; if there is more than one match,
+the parse is ambiguous, while if there are no matches at all, there is a syntax
+error.
+For more information about the macro parser's implementation, see the comments
+in src/librustc_expand/mbe/macro_parser.rs.
+
+There is an old and mostly undocumented effort to improve the MBE system, give
+it more hygiene-related features, better scoping and visibility rules, etc. There
+hasn't been a lot of work on this recently, unfortunately. Internally, macro
+macros use the same machinery as today's MBEs; they just have additional
+syntactic sugar and are allowed to be in namespaces.
+
+Precedural macros are also expanded during parsing, as mentioned above.
+However, they use a rather different mechanism. Rather than having a parser in
+the compiler, procedural macros are implemented as custom, third-party crates.
+The compiler will compile the proc macro crate and specially annotated
+functions in them (i.e. the proc macro itself), passing them a stream of tokens.
+The proc macro can then transform the token stream and output a new token
+stream, which is synthesized into the AST.
+It's worth noting that the token stream type used by proc macros is stable,
+so rustc does not use it internally (since our internal data structures are
+unstable). The compiler's token stream is
+rustc_ast::tokenstream::TokenStream, as previously. This is
+converted into the stable proc_macro::TokenStream and back in
+rustc_expand::proc_macro and rustc_expand::proc_macro_server.
+Because the Rust ABI is unstable, we use the C ABI for this conversion.
+TODO: more here.
+
+Custom derives are a special type of proc macro.
+TODO: more?
+
+In the previous chapters, we saw how the AST is built with all macros expanded.
+We saw how doing that requires doing some name resolution to resolve imports
+and macro names. In this chapter, we show how this is actually done and more.
+In fact, we don't do full name resolution during macro expansion -- we only
+resolve imports and macros at that time. This is required to know what to even
+expand. Later, after we have the whole AST, we due full name resolution to
+resolve all names in the crate. This happens in rustc_resolve::late.
+Unlike during macro expansion, in this late expansion, we only need to try to
+resolve a name once, since no new names can be added. If we fail to resolve a
+name now, then it is a compiler error.
+Name resolution can be complex. There are a few different namespaces (e.g.
+macros, values, types, lifetimes), and names my be valid at different (nested)
+scopes. Also, different types of names can fail to be resolved differently, and
+failures can happen differently at different scopes. For example, for a module
+scope, failure means no unexpanded macros and no unresolved glob imports in
+that module. On the other hand, in a function body, failure requires that a
+name be absent from the block we are in, all outer scopes, and the global
+scope.
+
+In our programs we can refer to variables, types, functions, etc, by giving them
+a name. These names are not always unique. For example, take this valid Rust
+program:
+
+#![allow(unused_variables)]
+fn main() {
+type x = u32;
+let x: x = 1;
+let y: x = 2;
+}
+
+How do we know on line 3 whether x is a type (u32) or a value (1)? These
+conflicts are resolved during name resolution. In this specific case, name
+resolution defines that type names and variable names live in separate
+namespaces and therefore can co-exist.
+The name resolution in Rust is a two-phase process. In the first phase, which runs
+during macro expansion, we build a tree of modules and resolve imports. Macro
+expansion and name resolution communicate with each other via the
+Resolver trait.
+The input to the second phase is the syntax tree, produced by parsing input
+files and expanding macros. This phase produces links from all the names in the
+source to relevant places where the name was introduced. It also generates
+helpful error messages, like typo suggestions, traits to import or lints about
+unused items.
+A successful run of the second phase (Resolver::resolve_crate) creates kind
+of an index the rest of the compilation may use to ask about the present names
+(through the hir::lowering::Resolver interface).
+The name resolution lives in the librustc_resolve crate, with the meat in
+lib.rs and some helpers or symbol-type specific logic in the other modules.
+
+Different kind of symbols live in different namespaces ‒ e.g. types don't
+clash with variables. This usually doesn't happen, because variables start with
+lower-case letter while types with upper case one, but this is only a
+convention. This is legal Rust code that'll compile (with warnings):
+
+#![allow(unused_variables)]
+fn main() {
+type x = u32;
+let x: x = 1;
+let y: x = 2; // See? x is still a type here.
+}
+
+To cope with this, and with slightly different scoping rules for these
+namespaces, the resolver keeps them separated and builds separate structures for
+them.
+In other words, when the code talks about namespaces, it doesn't mean the module
+hierarchy, it's types vs. values vs. macros.
+
+A name is visible only in certain area in the source code. This forms a
+hierarchical structure, but not necessarily a simple one ‒ if one scope is
+part of another, it doesn't mean the name visible in the outer one is also
+visible in the inner one, or that it refers to the same thing.
+To cope with that, the compiler introduces the concept of Ribs. This is
+abstraction of a scope. Every time the set of visible names potentially changes,
+a new rib is pushed onto a stack. The places where this can happen includes for
+example:
+
+- The obvious places ‒ curly braces enclosing a block, function boundaries,
+modules.
+- Introducing a let binding ‒ this can shadow another binding with the same
+name.
+- Macro expansion border ‒ to cope with macro hygiene.
+
+When searching for a name, the stack of ribs is traversed from the innermost
+outwards. This helps to find the closest meaning of the name (the one not
+shadowed by anything else). The transition to outer rib may also change the
+rules what names are usable ‒ if there are nested functions (not closures),
+the inner one can't access parameters and local bindings of the outer one,
+even though they should be visible by ordinary scoping rules. An example:
+
+#![allow(unused_variables)]
+fn main() {
+fn do_something<T: Default>(val: T) { // <- New rib in both types and values (1)
+ // `val` is accessible, as is the helper function
+ // `T` is accessible
+ let helper = || { // New rib on `helper` (2) and another on the block (3)
+ // `val` is accessible here
+ }; // End of (3)
+ // `val` is accessible, `helper` variable shadows `helper` function
+ fn helper() { // <- New rib in both types and values (4)
+ // `val` is not accessible here, (4) is not transparent for locals)
+ // `T` is not accessible here
+ } // End of (4)
+ let val = T::default(); // New rib (5)
+ // `val` is the variable, not the parameter here
+} // End of (5), (2) and (1)
+}
+
+Because the rules for different namespaces are a bit different, each namespace
+has its own independent rib stack that is constructed in parallel to the others.
+In addition, there's also a rib stack for local labels (e.g. names of loops or
+blocks), which isn't a full namespace in its own right.
+
+To perform the name resolution of the whole crate, the syntax tree is traversed
+top-down and every encountered name is resolved. This works for most kinds of
+names, because at the point of use of a name it is already introduced in the Rib
+hierarchy.
+There are some exceptions to this. Items are bit tricky, because they can be
+used even before encountered ‒ therefore every block needs to be first scanned
+for items to fill in its Rib.
+Other, even more problematic ones, are imports which need recursive fixed-point
+resolution and macros, that need to be resolved and expanded before the rest of
+the code can be processed.
+Therefore, the resolution is performed in multiple stages.
+
+This is a result of the first pass of learning the code. It is definitely
+incomplete and not detailed enough. It also might be inaccurate in places.
+Still, it probably provides useful first guidepost to what happens in there.
+
+- What exactly does it link to and how is that published and consumed by
+following stages of compilation?
+- Who calls it and how it is actually used.
+- Is it a pass and then the result is only used, or can it be computed
+incrementally (e.g. for RLS)?
+- The overall strategy description is a bit vague.
+- Where does the name
Rib come from?
+- Does this thing have its own tests, or is it tested only as part of some e2e
+testing?
+
+
+Today, rust programmers rely on a built in attribute called #[test]. All
+you have to do is mark a function as a test and include some asserts like so:
+#[test]
+fn my_test() {
+ assert!(2+2 == 4);
+}
+
+When this program is compiled using rustc --test or cargo test, it will
+produce an executable that can run this, and any other test function. This
+method of testing allows tests to live alongside code in an organic way. You
+can even put tests inside private modules:
+mod my_priv_mod {
+ fn my_priv_func() -> bool {}
+
+ #[test]
+ fn test_priv_func() {
+ assert!(my_priv_func());
+ }
+}
+
+Private items can thus be easily tested without worrying about how to expose
+them to any sort of external testing apparatus. This is key to the
+ergonomics of testing in Rust. Semantically, however, it's rather odd.
+How does any sort of main function invoke these tests if they're not visible?
+What exactly is rustc --test doing?
+#[test] is implemented as a syntactic transformation inside the compiler's
+librustc_ast crate. Essentially, it's a fancy macro, that
+rewrites the crate in 3 steps:
+
+As mentioned earlier, tests can exist inside private modules, so we need a
+way of exposing them to the main function, without breaking any existing
+code. To that end, librustc_ast will create local modules called
+__test_reexports that recursively reexport tests. This expansion translates
+the above example into:
+mod my_priv_mod {
+ fn my_priv_func() -> bool {}
+
+ pub fn test_priv_func() {
+ assert!(my_priv_func());
+ }
+
+ pub mod __test_reexports {
+ pub use super::test_priv_func;
+ }
+}
+
+Now, our test can be accessed as
+my_priv_mod::__test_reexports::test_priv_func. For deeper module
+structures, __test_reexports will reexport modules that contain tests, so a
+test at a::b::my_test becomes
+a::__test_reexports::b::__test_reexports::my_test. While this process seems
+pretty safe, what happens if there is an existing __test_reexports module?
+The answer: nothing.
+To explain, we need to understand how the AST represents
+identifiers. The name of every function, variable, module, etc. is
+not stored as a string, but rather as an opaque Symbol which is
+essentially an ID number for each identifier. The compiler keeps a separate
+hashtable that allows us to recover the human-readable name of a Symbol when
+necessary (such as when printing a syntax error). When the compiler generates
+the __test_reexports module, it generates a new Symbol for the identifier,
+so while the compiler-generated __test_reexports may share a name with your
+hand-written one, it will not share a Symbol. This technique prevents name
+collision during code generation and is the foundation of Rust's macro
+hygiene.
+
+Now that our tests are accessible from the root of our crate, we need to do
+something with them. librustc_ast generates a module like so:
+#[main]
+pub fn main() {
+ extern crate test;
+ test::test_main_static(&[&path::to::test1, /*...*/]);
+}
+
+where path::to::test1 is a constant of type test::TestDescAndFn.
+While this transformation is simple, it gives us a lot of insight into how
+tests are actually run. The tests are aggregated into an array and passed to
+a test runner called test_main_static. We'll come back to exactly what
+TestDescAndFn is, but for now, the key takeaway is that there is a crate
+called test that is part of Rust core, that implements all of the
+runtime for testing. test's interface is unstable, so the only stable way
+to interact with it is through the #[test] macro.
+
+If you've written tests in Rust before, you may be familiar with some of the
+optional attributes available on test functions. For example, a test can be
+annotated with #[should_panic] if we expect the test to cause a panic. It
+looks something like this:
+#[test]
+#[should_panic]
+fn foo() {
+ panic!("intentional");
+}
+
+This means our tests are more than just simple functions, they have
+configuration information as well. test encodes this configuration data
+into a struct called TestDesc. For each test function in a
+crate, librustc_ast will parse its attributes and generate a TestDesc
+instance. It then combines the TestDesc and test function into the
+predictably named TestDescAndFn struct, that test_main_static operates
+on. For a given test, the generated TestDescAndFn instance looks like so:
+self::test::TestDescAndFn{
+ desc: self::test::TestDesc{
+ name: self::test::StaticTestName("foo"),
+ ignore: false,
+ should_panic: self::test::ShouldPanic::Yes,
+ allow_fail: false,
+ },
+ testfn: self::test::StaticTestFn(||
+ self::test::assert_test_result(::crate::__test_reexports::foo())),
+}
+
+Once we've constructed an array of these test objects, they're passed to the
+test runner via the harness generated in step 2.
+
+On nightly rust, there's an unstable flag called unpretty that you can use
+to print out the module source after macro expansion:
+$ rustc my_mod.rs -Z unpretty=hir
+
+
+
+There are actually two panic macros - one defined in libcore, and one defined in libstd.
+This is due to the fact that code in libcore can panic. libcore is built before libstd,
+but we want panics to use the same machinery at runtime, whether they originate in libcore
+or libstd.
+
+The libcore panic! macro eventually makes the following call (in src/libcore/panicking.rs):
+
+#![allow(unused_variables)]
+fn main() {
+// NOTE This function never crosses the FFI boundary; it's a Rust-to-Rust call
+extern "Rust" {
+ #[lang = "panic_impl"]
+ fn panic_impl(pi: &PanicInfo<'_>) -> !;
+}
+
+let pi = PanicInfo::internal_constructor(Some(&fmt), location);
+unsafe { panic_impl(&pi) }
+}
+
+Actually resolving this goes through several layers of indirection:
+
+-
+
In src/librustc_middle/middle/weak_lang_items.rs, panic_impl is declared as 'weak lang item',
+with the symbol rust_begin_unwind. This is used in librustc_typeck/collect.rs
+to set the actual symbol name to rust_begin_unwind.
+Note that panic_impl is declared in an extern "Rust" block,
+which means that libcore will attempt to call a foreign symbol called rust_begin_unwind
+(to be resolved at link time)
+
+-
+
In src/libstd/panicking.rs, we have this definition:
+
+
+
+#![allow(unused_variables)]
+fn main() {
+/// Entry point of panic from the libcore crate.
+#[cfg(not(test))]
+#[panic_handler]
+#[unwind(allowed)]
+pub fn begin_panic_handler(info: &PanicInfo<'_>) -> ! {
+ ...
+}
+}
+
+The special panic_handler attribute is resolved via src/librustc_middle/middle/lang_items.
+The extract function converts the panic_handler attribute to a panic_impl lang item.
+Now, we have a matching panic_handler lang item in the libstd. This function goes
+through the same process as the extern { fn panic_impl } definition in libcore, ending
+up with a symbol name of rust_begin_unwind. At link time, the symbol reference in libcore
+will be resolved to the definition of libstd (the function called begin_panic_handler in the
+Rust source).
+Thus, control flow will pass from libcore to std at runtime. This allows panics from libcore
+to go through the same infrastructure that other panics use (panic hooks, unwinding, etc)
+
+This is where the actual panic-related logic begins. In src/libstd/panicking.rs,
+control passes to rust_panic_with_hook. This method is responsible
+for invoking the global panic hook, and checking for double panics. Finally,
+we call __rust_start_panic, which is provided by the panic runtime.
+The call to __rust_start_panic is very weird - it is passed a *mut &mut dyn BoxMeUp,
+converted to an usize. Let's break this type down:
+
+-
+
BoxMeUp is an internal trait. It is implemented for PanicPayload
+(a wrapper around the user-supplied payload type), and has a method
+fn box_me_up(&mut self) -> *mut (dyn Any + Send).
+This method takes the user-provided payload (T: Any + Send),
+boxes it, and converts the box to a raw pointer.
+
+-
+
When we call __rust_start_panic, we have an &mut dyn BoxMeUp.
+However, this is a fat pointer (twice the size of a usize).
+To pass this to the panic runtime across an FFI boundary, we take a mutable
+reference to this mutable reference (&mut &mut dyn BoxMeUp), and convert it to a raw pointer
+(*mut &mut dyn BoxMeUp). The outer raw pointer is a thin pointer, since it points to a Sized
+type (a mutable reference). Therefore, we can convert this thin pointer into a usize, which
+is suitable for passing across an FFI boundary.
+
+
+Finally, we call __rust_start_panic with this usize. We have now entered the panic runtime.
+
+Rust provides two panic runtimes: libpanic_abort and libpanic_unwind. The user chooses
+between them at build time via their Cargo.toml
+libpanic_abort is extremely simple: its implementation of __rust_start_panic just aborts,
+as you would expect.
+libpanic_unwind is the more interesting case.
+In its implementation of __rust_start_panic, we take the usize, convert
+it back to a *mut &mut dyn BoxMeUp, dereference it, and call box_me_up
+on the &mut dyn BoxMeUp. At this point, we have a raw pointer to the payload
+itself (a *mut (dyn Send + Any)): that is, a raw pointer to the actual value
+provided by the user who called panic!.
+At this point, the platform-independent code ends. We now call into
+platform-specific unwinding logic (e.g libunwind). This code is
+responsible for unwinding the stack, running any 'landing pads' associated
+with each frame (currently, running destructors), and transferring control
+to the catch_unwind frame.
+Note that all panics either abort the process or get caught by some call to catch_unwind:
+in src/libstd/rt.rs, the call to the user-provided main function is wrapped in catch_unwind.
+
+AST validation is the process of checking various correctness properties about
+the AST after macro expansion.
+TODO: write this chapter.
+
+TODO: this chapter
+
+The HIR – "High-Level Intermediate Representation" – is the primary IR used
+in most of rustc. It is a compiler-friendly representation of the abstract
+syntax tree (AST) that is generated after parsing, macro expansion, and name
+resolution (see Lowering for how the HIR is created).
+Many parts of HIR resemble Rust surface syntax quite closely, with
+the exception that some of Rust's expression forms have been desugared away.
+For example, for loops are converted into a loop and do not appear in
+the HIR. This makes HIR more amenable to analysis than a normal AST.
+This chapter covers the main concepts of the HIR.
+You can view the HIR representation of your code by passing the
+-Zunpretty=hir-tree flag to rustc:
+cargo rustc -- -Zunpretty=hir-tree
+
+
+The top-level data-structure in the HIR is the Crate, which stores
+the contents of the crate currently being compiled (we only ever
+construct HIR for the current crate). Whereas in the AST the crate
+data structure basically just contains the root module, the HIR
+Crate structure contains a number of maps and other things that
+serve to organize the content of the crate for easier access.
+For example, the contents of individual items (e.g. modules,
+functions, traits, impls, etc) in the HIR are not immediately
+accessible in the parents. So, for example, if there is a module item
+foo containing a function bar():
+
+#![allow(unused_variables)]
+fn main() {
+mod foo {
+ fn bar() { }
+}
+}
+
+then in the HIR the representation of module foo (the Mod
+struct) would only have the ItemId I of bar(). To get the
+details of the function bar(), we would lookup I in the
+items map.
+One nice result from this representation is that one can iterate
+over all items in the crate by iterating over the key-value pairs
+in these maps (without the need to trawl through the whole HIR).
+There are similar maps for things like trait items and impl items,
+as well as "bodies" (explained below).
+The other reason to set up the representation this way is for better
+integration with incremental compilation. This way, if you gain access
+to an &rustc_hir::Item (e.g. for the mod foo), you do not immediately
+gain access to the contents of the function bar(). Instead, you only
+gain access to the id for bar(), and you must invoke some
+function to lookup the contents of bar() given its id; this gives
+the compiler a chance to observe that you accessed the data for
+bar(), and then record the dependency.
+
+
+Most of the code that has to deal with things in HIR tends not to
+carry around references into the HIR, but rather to carry around
+identifier numbers (or just "ids"). Right now, you will find four
+sorts of identifiers in active use:
+
+DefId, which primarily names "definitions" or top-level items.
+
+- You can think of a
DefId as being shorthand for a very explicit
+and complete path, like std::collections::HashMap. However,
+these paths are able to name things that are not nameable in
+normal Rust (e.g. impls), and they also include extra information
+about the crate (such as its version number, as two versions of
+the same crate can co-exist).
+- A
DefId really consists of two parts, a CrateNum (which
+identifies the crate) and a DefIndex (which indexes into a list
+of items that is maintained per crate).
+
+HirId, which combines the index of a particular item with an
+offset within that item.
+
+- the key point of a
HirId is that it is relative to some item
+(which is named via a DefId).
+
+BodyId, this is an identifier that refers to a specific
+body (definition of a function or constant) in the crate. It is currently
+effectively a "newtype'd" HirId.NodeId, which is an absolute id that identifies a single node in the HIR
+tree.
+
+- While these are still in common use, they are being slowly phased out.
+- Since they are absolute within the crate, adding a new node anywhere in the
+tree causes the
NodeIds of all subsequent code in the crate to change.
+This is terrible for incremental compilation, as you can perhaps imagine.
+
+
+We also have an internal map to go from DefId to what’s called "Def path". "Def path" is like a
+module path but a bit more rich. For example, it may be crate::foo::MyStruct that identifies
+this definition uniquely. It’s a bit different than a module path because it might include a type
+parameter T, which you can't write in normal rust, like crate::foo::MyStruct::T. These are used
+in incremental compilation.
+
+Most of the time when you are working with the HIR, you will do so via
+the HIR Map, accessible in the tcx via tcx.hir_map (and defined in
+the hir::map module). The HIR map contains a number of methods to
+convert between IDs of various kinds and to lookup data associated
+with an HIR node.
+For example, if you have a DefId, and you would like to convert it
+to a NodeId, you can use
+tcx.hir.as_local_node_id(def_id). This returns
+an Option<NodeId> – this will be None if the def-id refers to
+something outside of the current crate (since then it has no HIR
+node), but otherwise returns Some(n) where n is the node-id of the
+definition.
+Similarly, you can use tcx.hir.find(n) to lookup the node for a
+NodeId. This returns a Option<Node<'tcx>>, where Node is an enum
+defined in the map; by matching on this you can find out what sort of
+node the node-id referred to and also get a pointer to the data
+itself. Often, you know what sort of node n is – e.g. if you know
+that n must be some HIR expression, you can do
+tcx.hir.expect_expr(n), which will extract and return the
+&hir::Expr, panicking if n is not in fact an expression.
+Finally, you can use the HIR map to find the parents of nodes, via
+calls like tcx.hir.get_parent_node(n).
+
+A rustc_hir::Body represents some kind of executable code, such as the body
+of a function/closure or the definition of a constant. Bodies are
+associated with an owner, which is typically some kind of item
+(e.g. an fn() or const), but could also be a closure expression
+(e.g. |x, y| x + y). You can use the HIR map to find the body
+associated with a given def-id (maybe_body_owned_by) or to find
+the owner of a body (body_owner_def_id).
+
+The lowering step converts AST to HIR.
+This means many structures are removed if they are irrelevant
+for type analysis or similar syntax agnostic analyses. Examples
+of such structures include but are not limited to
+
+- Parenthesis
+
+- Removed without replacement, the tree structure makes order explicit
+
+
+for loops and while (let) loops
+
+- Converted to
loop + match and some let bindings
+
+if let
+
+- Universal
impl Trait
+
+- Converted to generic arguments
+(but with some flags, to know that the user didn't write them)
+
+
+- Existential
impl Trait
+
+- Converted to a virtual
existential type declaration
+
+
+
+Lowering needs to uphold several invariants in order to not trigger the
+sanity checks in src/librustc_middle/hir/map/hir_id_validator.rs:
+
+- A
HirId must be used if created. So if you use the lower_node_id,
+you must use the resulting NodeId or HirId (either is fine, since
+any NodeIds in the HIR are checked for existing HirIds)
+- Lowering a
HirId must be done in the scope of the owning item.
+This means you need to use with_hir_id_owner if you are creating parts
+of an item other than the one being currently lowered. This happens for
+example during the lowering of existential impl Trait
+- A
NodeId that will be placed into a HIR structure must be lowered,
+even if its HirId is unused. Calling
+let _ = self.lower_node_id(node_id); is perfectly legitimate.
+- If you are creating new nodes that didn't exist in the
AST, you must
+create new ids for them. This is done by calling the next_id method,
+which produces both a new NodeId as well as automatically lowering it
+for you so you also get the HirId.
+
+If you are creating new DefIds, since each DefId needs to have a
+corresponding NodeId, it is advisable to add these NodeIds to the
+AST so you don't have to generate new ones during lowering. This has
+the advantage of creating a way to find the DefId of something via its
+NodeId. If lowering needs this DefId in multiple places, you can't
+generate a new NodeId in all those places because you'd also get a new
+DefId then. With a NodeId from the AST this is not an issue.
+Having the NodeId also allows the DefCollector to generate the DefIds
+instead of lowering having to do it on the fly. Centralizing the DefId
+generation in one place makes it easier to refactor and reason about.
+
+The -Zunpretty=hir-tree flag will dump out the HIR.
+If you are trying to correlate NodeIds or DefIds with source code, the
+--pretty expanded,identified flag may be useful.
+TODO: anything else?
+
+MIR is Rust's Mid-level Intermediate Representation. It is
+constructed from HIR. MIR was introduced in
+RFC 1211. It is a radically simplified form of Rust that is used for
+certain flow-sensitive safety checks – notably the borrow checker! –
+and also for optimization and code generation.
+If you'd like a very high-level introduction to MIR, as well as some
+of the compiler concepts that it relies on (such as control-flow
+graphs and desugaring), you may enjoy the
+rust-lang blog post that introduced MIR.
+
+MIR is defined in the src/librustc_middle/mir/ module, but much of the code
+that manipulates it is found in src/librustc_mir.
+Some of the key characteristics of MIR are:
+
+- It is based on a control-flow graph.
+- It does not have nested expressions.
+- All types in MIR are fully explicit.
+
+
+This section introduces the key concepts of MIR, summarized here:
+
+- Basic blocks: units of the control-flow graph, consisting of:
+
+- statements: actions with one successor
+- terminators: actions with potentially multiple successors; always at
+the end of a block
+- (if you're not familiar with the term basic block, see the background
+chapter)
+
+
+- Locals: Memory locations allocated on the stack (conceptually, at
+least), such as function arguments, local variables, and
+temporaries. These are identified by an index, written with a
+leading underscore, like
_1. There is also a special "local"
+(_0) allocated to store the return value.
+- Places: expressions that identify a location in memory, like
_1 or
+_1.f.
+- Rvalues: expressions that produce a value. The "R" stands for
+the fact that these are the "right-hand side" of an assignment.
+
+- Operands: the arguments to an rvalue, which can either be a
+constant (like
22) or a place (like _1).
+
+
+
+You can get a feeling for how MIR is structed by translating simple
+programs into MIR and reading the pretty printed output. In fact, the
+playground makes this easy, since it supplies a MIR button that will
+show you the MIR for your program. Try putting this program into play
+(or clicking on this link), and then clicking the "MIR"
+button on the top:
+fn main() {
+ let mut vec = Vec::new();
+ vec.push(1);
+ vec.push(2);
+}
+
+You should see something like:
+// WARNING: This output format is intended for human consumers only
+// and is subject to change without notice. Knock yourself out.
+fn main() -> () {
+ ...
+}
+
+This is the MIR format for the main function.
+Variable declarations. If we drill in a bit, we'll see it begins
+with a bunch of variable declarations. They look like this:
+let mut _0: (); // return place
+let mut _1: std::vec::Vec<i32>; // in scope 0 at src/main.rs:2:9: 2:16
+let mut _2: ();
+let mut _3: &mut std::vec::Vec<i32>;
+let mut _4: ();
+let mut _5: &mut std::vec::Vec<i32>;
+
+You can see that variables in MIR don't have names, they have indices,
+like _0 or _1. We also intermingle the user's variables (e.g.,
+_1) with temporary values (e.g., _2 or _3). You can tell apart
+user-defined variables because they have debuginfo associated to them (see below).
+User variable debuginfo. Below the variable declarations, we find the only
+hint that _1 represents a user variable:
+scope 1 {
+ debug vec => _1; // in scope 1 at src/main.rs:2:9: 2:16
+}
+
+Each debug <Name> => <Place>; annotation describes a named user variable,
+and where (i.e. the place) a debugger can find the data of that variable.
+Here the mapping is trivial, but optimizations may complicate the place,
+or lead to multiple user variables sharing the same place.
+Additionally, closure captures are described using the same system, and so
+they're complicated even without optimizations, e.g.: debug x => (*((*_1).0: &T));.
+The "scope" blocks (e.g., scope 1 { .. }) describe the lexical structure of
+the source program (which names were in scope when), so any part of the program
+annotated with // in scope 0 would be missing vec, if you were stepping
+through the code in a debugger, for example.
+Basic blocks. Reading further, we see our first basic block (naturally
+it may look slightly different when you view it, and I am ignoring some of the
+comments):
+bb0: {
+ StorageLive(_1);
+ _1 = const <std::vec::Vec<T>>::new() -> bb2;
+}
+
+A basic block is defined by a series of statements and a final
+terminator. In this case, there is one statement:
+StorageLive(_1);
+
+This statement indicates that the variable _1 is "live", meaning
+that it may be used later – this will persist until we encounter a
+StorageDead(_1) statement, which indicates that the variable _1 is
+done being used. These "storage statements" are used by LLVM to
+allocate stack space.
+The terminator of the block bb0 is the call to Vec::new:
+_1 = const <std::vec::Vec<T>>::new() -> bb2;
+
+Terminators are different from statements because they can have more
+than one successor – that is, control may flow to different
+places. Function calls like the call to Vec::new are always
+terminators because of the possibility of unwinding, although in the
+case of Vec::new we are able to see that indeed unwinding is not
+possible, and hence we list only one successor block, bb2.
+If we look ahead to bb2, we will see it looks like this:
+bb2: {
+ StorageLive(_3);
+ _3 = &mut _1;
+ _2 = const <std::vec::Vec<T>>::push(move _3, const 1i32) -> [return: bb3, unwind: bb4];
+}
+
+Here there are two statements: another StorageLive, introducing the _3
+temporary, and then an assignment:
+_3 = &mut _1;
+
+Assignments in general have the form:
+<Place> = <Rvalue>
+
+A place is an expression like _3, _3.f or *_3 – it denotes a
+location in memory. An Rvalue is an expression that creates a
+value: in this case, the rvalue is a mutable borrow expression, which
+looks like &mut <Place>. So we can kind of define a grammar for
+rvalues like so:
+<Rvalue> = & (mut)? <Place>
+ | <Operand> + <Operand>
+ | <Operand> - <Operand>
+ | ...
+
+<Operand> = Constant
+ | copy Place
+ | move Place
+
+As you can see from this grammar, rvalues cannot be nested – they can
+only reference places and constants. Moreover, when you use a place,
+we indicate whether we are copying it (which requires that the
+place have a type T where T: Copy) or moving it (which works
+for a place of any type). So, for example, if we had the expression x = a + b + c in Rust, that would get compiled to two statements and a
+temporary:
+TMP1 = a + b
+x = TMP1 + c
+
+(Try it and see, though you may want to do release mode to skip
+over the overflow checks.)
+
+The MIR data types are defined in the src/librustc_middle/mir/
+module. Each of the key concepts mentioned in the previous section
+maps in a fairly straightforward way to a Rust type.
+The main MIR data type is Mir. It contains the data for a single
+function (along with sub-instances of Mir for "promoted constants",
+but you can read about those below).
+
+- Basic blocks: The basic blocks are stored in the field
+
basic_blocks; this is a vector of BasicBlockData
+structures. Nobody ever references a basic block directly: instead,
+we pass around BasicBlock values, which are
+newtype'd indices into this vector.
+- Statements are represented by the type
Statement.
+- Terminators are represented by the
Terminator.
+- Locals are represented by a newtype'd index type
Local. The
+data for a local variable is found in the Mir (the local_decls
+vector). There is also a special constant RETURN_PLACE identifying
+the special "local" representing the return value.
+- Places are identified by the enum
Place. There are a few variants:
+
+- Local variables like
_1
+- Static variables
FOO
+- Projections, which are fields or other things that "project
+out" from a base place. So e.g. the place
_1.f is a projection,
+with f being the "projection element and _1 being the base
+path. *_1 is also a projection, with the * being represented
+by the ProjectionElem::Deref element.
+
+
+- Rvalues are represented by the enum
Rvalue.
+- Operands are represented by the enum
Operand.
+
+
+to be written
+
+
+to be written
+
+The lowering of HIR to MIR occurs for the following (probably incomplete)
+list of items:
+
+- Function and Closure bodies
+- Initializers of
static and const items
+- Initializers of enum discriminants
+- Glue and Shims of any kind
+
+- Tuple struct initializer functions
+- Drop code (the
Drop::drop function is not called directly)
+- Drop implementations of types without an explicit
Drop implementation
+
+
+
+The lowering is triggered by calling the mir_built query.
+There is an intermediate representation
+between HIR and MIR called the HAIR that is only used during the lowering.
+The HAIR's most important feature is that the various adjustments (which happen
+without explicit syntax) like coercions, autoderef, autoref and overloaded method
+calls have become explicit casts, deref operations, reference expressions or
+concrete function calls.
+The HAIR has datatypes that mirror the HIR datatypes, but instead of e.g. -x
+being a hair::ExprKind::Neg(hair::Expr) it is a hair::ExprKind::Neg(hir::Expr).
+This shallowness enables the HAIR to represent all datatypes that HIR has, but
+without having to create an in-memory copy of the entire HIR.
+MIR lowering will first convert the topmost expression from
+HIR to HAIR (in rustc_mir_build::hair::cx::expr) and then process
+the HAIR expressions recursively.
+The lowering creates local variables for every argument as specified in the signature.
+Next it creates local variables for every binding specified (e.g. (a, b): (i32, String))
+produces 3 bindings, one for the argument, and two for the bindings. Next it generates
+field accesses that read the fields from the argument and writes the value to the binding
+variable.
+With this initialization out of the way, the lowering triggers a recursive call
+to a function that generates the MIR for the body (a Block expression) and
+writes the result into the RETURN_PLACE.
+
+Functions that generate MIR tend to fall into one of two patterns.
+First, if the function generates only statements, then it will take a
+basic block as argument onto which those statements should be appended.
+It can then return a result as normal:
+fn generate_some_mir(&mut self, block: BasicBlock) -> ResultType {
+ ...
+}
+
+But there are other functions that may generate new basic blocks as well.
+For example, lowering an expression like if foo { 22 } else { 44 }
+requires generating a small "diamond-shaped graph".
+In this case, the functions take a basic block where their code starts
+and return a (potentially) new basic block where the code generation ends.
+The BlockAnd type is used to represent this:
+fn generate_more_mir(&mut self, block: BasicBlock) -> BlockAnd<ResultType> {
+ ...
+}
+
+When you invoke these functions, it is common to have a local variable block
+that is effectively a "cursor". It represents the point at which we are adding new MIR.
+When you invoke generate_more_mir, you want to update this cursor.
+You can do this manually, but it's tedious:
+let mut block;
+let v = match self.generate_more_mir(..) {
+ BlockAnd { block: new_block, value: v } => {
+ block = new_block;
+ v
+ }
+};
+
+For this reason, we offer a macro that lets you write
+let v = unpack!(block = self.generate_more_mir(...)).
+It simply extracts the new block and overwrites the
+variable block that you named in the unpack!.
+
+There are essentially four kinds of representations one might want of an expression:
+
+Place refers to a (or part of a) preexisting memory location (local, static, promoted)Rvalue is something that can be assigned to a PlaceOperand is an argument to e.g. a + operation or a function call- a temporary variable containing a copy of the value
+
+The following image depicts a general overview of the interactions between the
+representations:
+
+Click here for a more detailed view
+We start out with lowering the function body to an Rvalue so we can create an
+assignment to RETURN_PLACE, This Rvalue lowering will in turn trigger lowering to
+Operand for its arguments (if any). Operand lowering either produces a const
+operand, or moves/copies out of a Place, thus triggering a Place lowering. An
+expression being lowered to a Place can in turn trigger a temporary to be created
+if the expression being lowered contains operations. This is where the snake bites its
+own tail and we need to trigger an Rvalue lowering for the expression to be written
+into the local.
+
+Operators on builtin types are not lowered to function calls (which would end up being
+infinite recursion calls, because the trait impls just contain the operation itself
+again). Instead there are Rvalues for binary and unary operators and index operations.
+These Rvalues later get codegened to llvm primitive operations or llvm intrinsics.
+Operators on all other types get lowered to a function call to their impl of the
+operator's corresponding trait.
+Regardless of the lowering kind, the arguments to the operator are lowered to Operands.
+This means all arguments are either constants, or refer to an already existing value
+somewhere in a local or static.
+
+Method calls are lowered to the same TerminatorKind that function calls are.
+In MIR there is no difference between method calls and function calls anymore.
+
+if conditions and match statements for enums without variants with fields are
+lowered to TerminatorKind::SwitchInt. Each possible value (so 0 and 1 for if
+conditions) has a corresponding BasicBlock to which the code continues.
+The argument being branched on is (again) an Operand representing the value of
+the if condition.
+
+match statements for enums with variants that have fields are lowered to
+TerminatorKind::SwitchInt, too, but the Operand refers to a Place where the
+discriminant of the value can be found. This often involves reading the discriminant
+to a new temporary variable.
+
+Aggregate values of any kind (e.g. structs or tuples) are built via Rvalue::Aggregate.
+All fields are
+lowered to Operators. This is essentially equivalent to one assignment
+statement per aggregate field plus an assignment to the discriminant in the
+case of enums.
+
+The MIR visitor is a convenient tool for traversing the MIR and either
+looking for things or making changes to it. The visitor traits are
+defined in the rustc::mir::visit module – there are two of
+them, generated via a single macro: Visitor (which operates on a
+&Mir and gives back shared references) and MutVisitor (which
+operates on a &mut Mir and gives back mutable references).
+To implement a visitor, you have to create a type that represents
+your visitor. Typically, this type wants to "hang on" to whatever
+state you will need while processing MIR:
+struct MyVisitor<...> {
+ tcx: TyCtxt<'tcx>,
+ ...
+}
+
+and you then implement the Visitor or MutVisitor trait for that type:
+impl<'tcx> MutVisitor<'tcx> for NoLandingPads {
+ fn visit_foo(&mut self, ...) {
+ ...
+ self.super_foo(...);
+ }
+}
+
+As shown above, within the impl, you can override any of the
+visit_foo methods (e.g., visit_terminator) in order to write some
+code that will execute whenever a foo is found. If you want to
+recursively walk the contents of the foo, you then invoke the
+super_foo method. (NB. You never want to override super_foo.)
+A very simple example of a visitor can be found in NoLandingPads.
+That visitor doesn't even require any state: it just visits all
+terminators and removes their unwind successors.
+
+In addition the visitor, the rustc::mir::traversal module
+contains useful functions for walking the MIR CFG in
+different standard orders (e.g. pre-order, reverse
+post-order, and so forth).
+
+If you would like to get the MIR for a function (or constant, etc),
+you can use the optimized_mir(def_id) query. This will give you back
+the final, optimized MIR. For foreign def-ids, we simply read the MIR
+from the other crate's metadata. But for local def-ids, the query will
+construct the MIR and then iteratively optimize it by applying a
+series of passes. This section describes how those passes work and how
+you can extend them.
+To produce the optimized_mir(D) for a given def-id D, the MIR
+passes through several suites of optimizations, each represented by a
+query. Each suite consists of multiple optimizations and
+transformations. These suites represent useful intermediate points
+where we want to access the MIR for type checking or other purposes:
+
+mir_build(D) – not a query, but this constructs the initial MIRmir_const(D) – applies some simple transformations to make MIR ready for
+constant evaluation;mir_validated(D) – applies some more transformations, making MIR ready for
+borrow checking;optimized_mir(D) – the final state, after all optimizations have been
+performed.
+
+A MirPass is some bit of code that processes the MIR, typically –
+but not always – transforming it along the way somehow. For example,
+it might perform an optimization. The MirPass trait itself is found
+in the rustc_mir::transform module, and it
+basically consists of one method, run_pass, that simply gets an
+&mut Mir (along with the tcx and some information about where it
+came from). The MIR is therefore modified in place (which helps to
+keep things efficient).
+A good example of a basic MIR pass is NoLandingPads, which walks
+the MIR and removes all edges that are due to unwinding – this is
+used when configured with panic=abort, which never unwinds. As you
+can see from its source, a MIR pass is defined by first defining a
+dummy type, a struct with no fields, something like:
+
+#![allow(unused_variables)]
+fn main() {
+struct MyPass;
+}
+
+for which you then implement the MirPass trait. You can then insert
+this pass into the appropriate list of passes found in a query like
+optimized_mir, mir_validated, etc. (If this is an optimization, it
+should go into the optimized_mir list.)
+If you are writing a pass, there's a good chance that you are going to
+want to use a MIR visitor. MIR visitors are a handy way to walk all
+the parts of the MIR, either to search for something or to make small
+edits.
+
+The intermediate queries mir_const() and mir_validated() yield up
+a &'tcx Steal<Mir<'tcx>>, allocated using
+tcx.alloc_steal_mir(). This indicates that the result may be
+stolen by the next suite of optimizations – this is an
+optimization to avoid cloning the MIR. Attempting to use a stolen
+result will cause a panic in the compiler. Therefore, it is important
+that you do not read directly from these intermediate queries except as
+part of the MIR processing pipeline.
+Because of this stealing mechanism, some care must also be taken to
+ensure that, before the MIR at a particular phase in the processing
+pipeline is stolen, anyone who may want to read from it has already
+done so. Concretely, this means that if you have some query foo(D)
+that wants to access the result of mir_const(D) or
+mir_validated(D), you need to have the successor pass "force"
+foo(D) using ty::queries::foo::force(...). This will force a query
+to execute even though you don't directly require its result.
+As an example, consider MIR const qualification. It wants to read the
+result produced by the mir_const() suite. However, that result will
+be stolen by the mir_validated() suite. If nothing was done,
+then mir_const_qualif(D) would succeed if it came before
+mir_validated(D), but fail otherwise. Therefore, mir_validated(D)
+will force mir_const_qualif before it actually steals, thus
+ensuring that the reads have already happened (remember that
+queries are memoized, so executing a query twice
+simply loads from a cache the second time):
+mir_const(D) --read-by--> mir_const_qualif(D)
+ | ^
+ stolen-by |
+ | (forces)
+ v |
+mir_validated(D) ------------+
+
+This mechanism is a bit dodgy. There is a discussion of more elegant
+alternatives in rust-lang/rust#41710.
+
+This section describes how rustc handles closures. Closures in Rust are
+effectively "desugared" into structs that contain the values they use (or
+references to the values they use) from their creator's stack frame. rustc has
+the job of figuring out which values a closure uses and how, so it can decide
+whether to capture a given variable by shared reference, mutable reference, or
+by move. rustc also has to figure out which the closure traits (Fn,
+FnMut, or FnOnce) a closure is capable of
+implementing.
+Let's start with a few examples:
+
+To start, let's take a look at how the closure in the following example is desugared:
+fn closure(f: impl Fn()) {
+ f();
+}
+
+fn main() {
+ let x: i32 = 10;
+ closure(|| println!("Hi {}", x)); // The closure just reads x.
+ println!("Value of x after return {}", x);
+}
+
+Let's say the above is the content of a file called immut.rs. If we compile
+immut.rs using the following command. The -Zdump-mir=all flag will cause
+rustc to generate and dump the MIR to a directory called mir_dump.
+> rustc +stage1 immut.rs -Zdump-mir=all
+
+After we run this command, we will see a newly generated directory in our
+current working directory called mir_dump, which will contain several files.
+If we look at file rustc.main.-------.mir_map.0.mir, we will find, among
+other things, it also contains this line:
+_4 = &_1;
+_3 = [closure@immut.rs:7:13: 7:36] { x: move _4 };
+
+Note that in the MIR examples in this chapter, _1 is x.
+Here in first line _4 = &_1;, the mir_dump tells us that x was borrowed
+as an immutable reference. This is what we would hope as our closure just
+reads x.
+
+Here is another example:
+fn closure(mut f: impl FnMut()) {
+ f();
+}
+
+fn main() {
+ let mut x: i32 = 10;
+ closure(|| {
+ x += 10; // The closure mutates the value of x
+ println!("Hi {}", x)
+ });
+ println!("Value of x after return {}", x);
+}
+
+_4 = &mut _1;
+_3 = [closure@mut.rs:7:13: 10:6] { x: move _4 };
+
+This time along, in the line _4 = &mut _1;, we see that the borrow is changed to mutable borrow.
+Fair enough! The closure increments x by 10.
+
+One more example:
+fn closure(f: impl FnOnce()) {
+ f();
+}
+
+fn main() {
+ let x = vec![21];
+ closure(|| {
+ drop(x); // Makes x unusable after the fact.
+ });
+ // println!("Value of x after return {:?}", x);
+}
+
+_6 = [closure@move.rs:7:13: 9:6] { x: move _1 }; // bb16[3]: scope 1 at move.rs:7:13: 9:6
+
+Here, x is directly moved into the closure and the access to it will not be permitted after the
+closure.
+
+Now let's dive into rustc code and see how all these inferences are done by the compiler.
+Let's start with defining a term that we will be using quite a bit in the rest of the discussion -
+upvar. An upvar is a variable that is local to the function where the closure is defined. So,
+in the above examples, x will be an upvar to the closure. They are also sometimes referred to as
+the free variables meaning they are not bound to the context of the closure.
+src/librustc_middle/ty/query/mod.rs defines a query called upvars_mentioned
+for this purpose.
+Other than lazy invocation, one other thing that the distinguishes a closure from a
+normal function is that it can use the upvars. It borrows these upvars from its surrounding
+context; therefore the compiler has to determine the upvar's borrow type. The compiler starts with
+assigning an immutable borrow type and lowers the restriction (that is, changes it from
+immutable to mutable to move) as needed, based on the usage. In the Example 1 above, the
+closure only uses the variable for printing but does not modify it in any way and therefore, in the
+mir_dump, we find the borrow type for the upvar x to be immutable. In example 2, however, the
+closure modifies x and increments it by some value. Because of this mutation, the compiler, which
+started off assigning x as an immutable reference type, has to adjust it as a mutable reference.
+Likewise in the third example, the closure drops the vector and therefore this requires the variable
+x to be moved into the closure. Depending on the borrow kind, the closure has to implement the
+appropriate trait: Fn trait for immutable borrow, FnMut for mutable borrow,
+and FnOnce for move semantics.
+Most of the code related to the closure is in the
+src/librustc_typeck/check/upvar.rs file and the data structures are
+declared in the file src/librustc_middle/ty/mod.rs.
+Before we go any further, let's discuss how we can examine the flow of control through the rustc
+codebase. For closures specifically, set the RUST_LOG env variable as below and collect the
+output in a file:
+> RUST_LOG=rustc_typeck::check::upvar rustc +stage1 -Zdump-mir=all \
+ <.rs file to compile> 2> <file where the output will be dumped>
+
+This uses the stage1 compiler and enables debug! logging for the
+rustc_typeck::check::upvar module.
+The other option is to step through the code using lldb or gdb.
+
+rust-lldb build/x86_64-apple-darwin/stage1/bin/rustc test.rs- In lldb:
+
+b upvar.rs:134 // Setting the breakpoint on a certain line in the upvar.rs file`r // Run the program until it hits the breakpoint
+
+
+Let's start with upvar.rs. This file has something called
+the euv::ExprUseVisitor which walks the source of the closure and
+invokes a callbackfor each upvar that is borrowed, mutated, or moved.
+fn main() {
+ let mut x = vec![21];
+ let _cl = || {
+ let y = x[0]; // 1.
+ x[0] += 1; // 2.
+ };
+}
+
+In the above example, our visitor will be called twice, for the lines marked 1 and 2, once for a
+shared borrow and another one for a mutable borrow. It will also tell us what was borrowed.
+The callbacks are defined by implementing the Delegate trait. The
+InferBorrowKind type implements Delegate and keeps a map that
+records for each upvar which mode of borrow was required. The modes of borrow
+can be ByValue (moved) or ByRef (borrowed). For ByRef borrows, it can be
+shared, shallow, unique or mut as defined in the
+src/librustc_middle/mir/mod.rs.
+Delegate defines a few different methods (the different callbacks):
+consume: for move of a variable, borrow for a borrow of some kind
+(shared or mutable), and mutate when we see an assignment of something.
+All of these callbacks have a common argument cmt which stands for Category,
+Mutability and Type and is defined in
+src/librustc_middle/middle/mem_categorization.rs. Borrowing from the code
+comments, "cmt is a complete categorization of a value indicating where it
+originated and how it is located, as well as the mutability of the memory in
+which the value is stored". Based on the callback (consume, borrow etc.), we
+will call the relevant adjust_upvar_borrow_kind_for_ and pass the
+cmt along. Once the borrow type is adjusted, we store it in the table, which
+basically says what borrows were made for each closure.
+self.tables
+ .borrow_mut()
+ .upvar_capture_map
+ .extend(delegate.adjust_upvar_captures);
+
+
+This part discusses the many analyses that the compiler uses to check various
+properties of the code and to inform later stages. Typically, this is what people
+mean when they talk about "Rust's type system". This includes the
+representation, inference, and checking of types, the trait system, and the
+borrow checker. These analyses do not happen as one big pass or set of
+contiguous passes. Rather, they are spread out throughout various parts of the
+compilation process and use different intermediate representations. For example,
+type checking happens on the HIR, while borrow checking happens on the MIR.
+Nonetheless, for the sake of presentation, we will discuss all of these
+analyses in this part of the guide.
+
+The ty module defines how the Rust compiler represents types internally. It also defines the
+typing context (tcx or TyCtxt), which is the central data structure in the compiler.
+
+When we talk about how rustc represents types, we usually refer to a type called Ty . There are
+quite a few modules and types for Ty in the compiler (Ty documentation).
+The specific Ty we are referring to is rustc::ty::Ty (and not
+rustc_hir::Ty). The distinction is important, so we will discuss it first before going
+into the details of ty::Ty.
+
+The HIR in rustc can be thought of as the high-level intermediate representation. It is more or less
+the AST (see this chapter) as it represents the
+syntax that the user wrote, and is obtained after parsing and some desugaring. It has a
+representation of types, but in reality it reflects more of what the user wrote, that is, what they
+wrote so as to represent that type.
+In contrast, ty::Ty represents the semantics of a type, that is, the meaning of what the user
+wrote. For example, rustc_hir::Ty would record the fact that a user used the name u32 twice
+in their program, but the ty::Ty would record the fact that both usages refer to the same type.
+Example: fn foo(x: u32) → u32 { } In this function we see that u32 appears twice. We know
+that that is the same type, i.e. the function takes an argument and returns an argument of the same
+type, but from the point of view of the HIR there would be two distinct type instances because these
+are occurring in two different places in the program. That is, they have two
+different Spans (locations).
+Example: fn foo(x: &u32) -> &u32) In addition, HIR might have information left out. This type
+&u32 is incomplete, since in the full rust type there is actually a lifetime, but we didn’t need
+to write those lifetimes. There are also some elision rules that insert information. The result may
+look like fn foo<'a>(x: &'a u32) -> &'a u32).
+In the HIR level, these things are not spelled out and you can say the picture is rather incomplete.
+However, at the ty::Ty level, these details are added and it is complete. Moreover, we will have
+exactly one ty::Ty for a given type, like u32, and that ty::Ty is used for all u32s in the
+whole program, not a specific usage, unlike rustc_hir::Ty.
+Here is a summary:
+There are a lot of related types, and we’ll cover them in time (e.g regions/lifetimes,
+“substitutions”, etc).
+Sometimes there is a third case. You believe that an error has been reported, but you believe it
+would've been reported earlier in the compilation, not locally. In that case, you can invoke
+delay_span_bug This will make a note that you expect compilation to yield an error -- if however
+compilation should succeed, then it will trigger a compiler bug report.
+Well, the alternate way we could have choosen to represent types would be to always create a new,
+fully-substituted form of the AdtDef where all the types are already substituted. This seems like
+less of a hassle. However, the (AdtDef, substs) scheme has some advantages over this.
+The index of the type parameter is an integer indicating its order in the list of the type
+parameters. Moreover, we consider the list to include all of the type parameters from outer scopes.
+Consider the following example:
+Let’s look a bit more closely at that last substitution to see why we use indexes. If we want to
+find the type of foo.x, we can get generic type of x, which is Vec<Param(0)>. Now we can take
+the index 0 and use it to find the right type substitution: looking at Foo's SubstsRef, we
+have the list [u32, f32] , since we want to replace index 0, we take the 0-th index of this
+list, which is u32. Voila!
+You can think of it with this analogy to the iterator combinators we have come to love in rust:
+In almost all cases, we don’t want to replace the whole struct; we only want to replace ty::Tys in
+the struct, so usually we call super_fold_with. A typical implementation that MyFoldable could
+have might do something like this:
+Type inference is the process of automatic detection of the type of an
+expression.
+It is what allows Rust to work with fewer or no type annotations,
+making things easier for users:
+The type inference is based on the standard Hindley-Milner (HM) type inference
+algorithm, but extended in various way to accommodate subtyping, region
+inference, and higher-ranked types.
+We use the terms "region" and "lifetime" interchangeably. Both refer to
+the 'a in &'a T.
+The term "bound region" refers to a region that is bound in a function
+signature, such as the 'a in for<'a> fn(&'a u32). A region is
+"free" if it is not bound.
+You create and "enter" an inference context by doing something like
+the following:
+If you're familiar with the basic ideas of unification from H-M type
+systems, or logic languages like Prolog, this is the same concept. If
+you're not, you might want to read a tutorial on how H-M type
+inference works, or perhaps this blog post on
+unification in the Chalk project.
+All told, the inference context stores four kinds of inference variables as of
+this writing:
+All the type variables work in much the same way: you can create a new
+type variable, and what you get is Ty<'tcx> representing an
+unresolved type ?T. Then later you can apply the various operations
+that the inferencer supports, such as equality or subtyping, and it
+will possibly instantiate (or bind) that ?T to a specific
+value as a result.
+The region variables work somewhat differently, and are described
+below in a separate section.
+When you equate things, you force them to be precisely equal. Equating
+returns an InferResult – if it returns Err(err), then equating
+failed, and the enclosing TypeError will tell you what went wrong.
+Rather than use snapshots directly, it is often helpful to use the
+methods like commit_if_ok or probe that encapsulate higher-level
+patterns.
+One thing worth discussing is subtyping obligations. When you force
+two types to be a subtype, like ?T <: i32, we can often convert those
+into equality constraints. This follows from Rust's rather limited notion
+of subtyping: so, in the above case, ?T <: i32 is equivalent to ?T = i32.
+However, in some cases we have to be more careful. For example, when
+regions are involved. So if you have ?T <: &'a i32, what we would do
+is to first "generalize" &'a i32 into a type with a region variable:
+&'?b i32, and then unify ?T with that (?T = &'?b i32). We then
+relate this new variable with the original bound:
+Regions are inferenced somewhat differently from types. Rather than
+eagerly unifying things, we simply collect constraints as we go, but
+make (almost) no attempt to solve regions. These constraints have the
+form of an "outlives" constraint:
+Actually the code tends to view them as a subregion relation, but it's the same
+idea:
+There is one case where we do some amount of eager unification. If you have an
+equality constraint between two regions
+Ultimately, region constraints are only solved at the very end of
+type-checking, once all other constraints are known. There are two
+ways to solve region constraints right now: lexical and
+non-lexical. Eventually there will only be one.
+Non-lexical region constraints are not handled within the inference
+context. Instead, the NLL solver (actually, the MIR type-checker)
+invokes take_and_reset_region_constraints periodically. This
+extracts all of the outlives constraints from the region solver, but
+leaves the set of variables intact. This is used to get just the
+region constraints that resulted from some particular point in the
+program, since the NLL solver needs to know not just what regions
+were subregions, but also where. Finally, the NLL solver invokes
+take_region_var_origins, which "closes" the region constraint
+process in the same way as normal solving.
+Lexical region resolution is done by initially assigning each region
+variable to an empty value. We then process each outlives constraint
+repeatedly, growing region variables until a fixed-point is reached.
+Region variables can be grown using a least-upper-bound relation on
+the region lattice in a fairly straightforward fashion.
+Trait resolution is the process of pairing up an impl with each
+reference to a trait. So, for example, if there is a generic function like:
+it is the job of trait resolution to figure out whether there exists an impl of
+(in this case) isize : Clone.
+Note that in some cases, like generic functions, we may not be able to
+find a specific impl, but we can figure out that the caller must
+provide an impl. For example, consider the body of clone_slice:
+During type checking, we do not store the results of trait selection.
+We simply wish to verify that trait selection will succeed. Then
+later, at trans time, when we have all concrete types available, we
+can repeat the trait selection to choose an actual implementation, which
+will then be generated in the output binary.
+Selection is the process of deciding whether an obligation can be
+resolved and, if so, how it is to be resolved (via impl, where clause, etc).
+The main interface is the select() function, which takes an obligation
+and returns a SelectionResult. There are three possible outcomes:
+The basic algorithm for selection is broken into two big phases:
+candidate assembly and confirmation.
+Note that because of how lifetime inference works, it is not possible to
+give back immediate feedback as to whether a unification or subtype
+relationship between lifetimes holds or not. Therefore, lifetime
+matching is not considered during selection. This is reflected in
+the fact that subregion assignment is infallible. This may yield
+lifetime constraints that will later be found to be in error (in
+contrast, the non-lifetime-constraints have already been checked
+during selection and can never cause an error, though naturally they
+may lead to other errors downstream).
+Searches for impls/where-clauses/etc that might
+possibly be used to satisfy the obligation. Each of those is called
+a candidate. To avoid ambiguity, we want to find exactly one
+candidate that is definitively applicable. In some cases, we may not
+know whether an impl/where-clause applies or not – this occurs when
+the obligation contains unbound inference variables.
+The subroutines that decide whether a particular impl/where-clause/etc
+applies to a particular obligation are collectively referred to as the
+process of matching. At the moment, this amounts to
+unifying the Self types, but in the future we may also recursively
+consider some of the nested obligations, in the case of an impl.
+The basic idea for candidate assembly is to do a first pass in which
+we identify all possible candidates. During this pass, all that we do
+is try and unify the type parameters. (In particular, we ignore any
+nested where clauses.) Presuming that this unification succeeds, the
+impl is added as a candidate.
+Once this first pass is done, we can examine the set of candidates. If
+it is a singleton set, then we are done: this is the only impl in
+scope that could possibly apply. Otherwise, we can winnow down the set
+of candidates by using where clauses and other conditions. If this
+reduced set yields a single, unambiguous entry, we're good to go,
+otherwise the result is considered ambiguous.
+This process is easier if we work through some examples. Consider
+the following trait:
+This trait just has one method. It's about as simple as it gets. It
+converts from the (implicit) Self type to the Target type. If we
+wanted to permit conversion between isize and usize, we might
+implement Convert like so:
+But what happens if there are multiple impls where all the types
+unify? Consider this example:
+Besides an impl, the other major way to resolve an obligation is via a
+where clause. The selection process is always given a parameter
+environment which contains a list of where clauses, which are
+basically obligations that we can assume are satisfiable. We will iterate
+over that list and check whether our current obligation can be found
+in that list. If so, it is considered satisfied. More precisely, we
+want to check whether there is a where-clause obligation that is for
+the same trait (or some subtrait) and which can match against the obligation.
+Confirmation is where an error would be reported because the impl specified
+that Target would be usize, but the obligation reported char. Hence the
+result of selection would be an error.
+As mentioned above, during type checking, we do not store the results of trait
+selection. At trans time, we repeat the trait selection to choose a particular
+impl for each method call. In this second selection, we do not consider any
+where-clauses to be in scope because we know that each resolution will resolve
+to a particular impl.
+One interesting twist has to do with nested obligations. In general, in trans,
+we only need to do a "shallow" selection for an obligation. That is, we wish to
+identify which impl applies, but we do not (yet) need to decide how to select
+any nested obligations. Nonetheless, we do currently do a complete resolution,
+and that is because it can sometimes inform the results of type inference.
+That is, we do not have the full substitutions in terms of the type variables
+of the impl available to us, so we must run trait selection to figure
+everything out.
+So let's work through our example.
+Let's consider a failure case. Imagine we also have a struct
+Once the basic matching is done, we get to another interesting topic:
+how to deal with impl obligations. I'll work through a simple example
+here. Imagine we have the traits Foo and Bar and an associated impl:
+After the matching, we are in a position where we have a placeholder
+substitution like X => &'0 isize. If we apply this substitution to the
+impl obligations, we get F : Bar<&'0 isize>. Obviously this is not
+directly usable because the placeholder region '0 cannot leak out of
+our computation.
+In general, we attempt to cache the results of trait selection. This
+is a somewhat complex process. Part of the reason for this is that we
+want to be able to cache results even when all the types in the trait
+reference are not fully known. In that case, it may happen that the
+trait selection process is also influencing type variables, so we have
+to be able to not only cache the result of the selection process,
+but replay its effects on the type variables.
+The high-level idea of how the cache works is that we first replace
+all unbound inference variables with placeholder versions. Therefore,
+if we had a trait reference usize : Foo<$t>, where $t is an unbound
+inference variable, we might replace it with usize : Foo<$0>, where
+$0 is a placeholder type. We would then look this up in the cache.
+If we found a hit, the hit would tell us the immediate next step to
+take in the selection process (e.g. apply impl #22, or apply where
+clause X : Foo<Y>).
+One subtle interaction is that the results of trait lookup will vary
+depending on what where clauses are in scope. Therefore, we actually
+have two caches, a local and a global cache. The local cache is
+attached to the ParamEnv, and the global cache attached to the
+tcx. We use the local cache whenever the result might depend on the
+where clauses that are in scope. The determination of which cache to
+use is done by the method pick_candidate_cache in select.rs. At
+the moment, we use a very simple, conservative rule: if there are any
+where-clauses in scope, then we use the local cache. We used to try
+and draw finer-grained distinctions, but that led to a serious of
+annoying and weird bugs like #22019 and #18290. This simple rule seems
+to be pretty clearly safe and also still retains a very high hit rate
+(~95% when compiling rustc).
+You might expect that the specialization graph would be used during
+selection – i.e. when actually performing specialization. This is
+not done for two reasons:
+Trait impl selection can succeed even when multiple impls can apply,
+as long as they are part of the same specialization family. In that
+case, it returns a single impl on success – this is the most
+specialized impl known to apply. However, if there are any inference
+variables in play, the returned impl may not be the actual impl we
+will use at trans time. Thus, we take special care to avoid projecting
+associated types unless either (1) the associated type does not use
+default and thus cannot be overridden or (2) all input types are
+known concretely.
+The key observation here is that the Rust trait system is basically a
+kind of logic, and it can be mapped onto standard logical inference
+rules. We can then look for solutions to those inference rules in a
+very similar fashion to how e.g. a Prolog solver works. It turns out
+that we can't quite use Prolog rules (also called Horn clauses) but
+rather need a somewhat more expressive variant.
+The key observation here is that the Rust trait system is basically a
+kind of logic, and it can be mapped onto standard logical inference
+rules. We can then look for solutions to those inference rules in a
+very similar fashion to how e.g. a Prolog solver works. It turns out
+that we can't quite use Prolog rules (also called Horn clauses) but
+rather need a somewhat more expressive variant.
+One of the first observations is that the Rust trait system is
+basically a kind of logic. As such, we can map our struct, trait, and
+impl declarations into logical inference rules. For the most part,
+these are basically Horn clauses, though we'll see that to capture the
+full richness of Rust – and in particular to support generic
+programming – we have to go a bit further than standard Horn clauses.
+To see how this mapping works, let's start with an example. Imagine
+we declare a trait and a few impls, like so:
+We could map these declarations to some Horn clauses, written in a
+Prolog-like notation, as follows:
+We can easily extend the example above to cover generic traits with
+more than one input type. So imagine the Eq<T> trait, which declares
+that Self is equatable with a value of type T:
+So far so good.
+OK, now that we have defined some logical rules that are able to
+express when traits are implemented and to handle associated types,
+let's turn our focus a bit towards type-checking. Type-checking is
+interesting because it is what gives us the goals that we need to
+prove. That is, everything we've seen so far has been about how we
+derive the rules by which we can prove goals from the traits and impls
+in the program; but we are also interested in how to derive the goals
+that we need to prove, and those come from type-checking.
+If we wanted, we could write a Prolog predicate that defines the
+conditions under which bar() can be called. We'll say that those
+conditions are called being "well-formed":
+Ok, so far so good. Let's move on to type-checking a more complex function.
+In the last section, we used standard Prolog horn-clauses (augmented with Rust's
+notion of type equality) to type-check some simple Rust functions. But that only
+works when we are type-checking non-generic functions. If we want to type-check
+a generic function, it turns out we need a stronger notion of goal than what Prolog
+can provide. To see what I'm talking about, let's revamp our previous
+example to make foo generic:
+This notation I'm using here is the notation I've been using in my
+prototype implementation; it's similar to standard mathematical
+notation but a bit Rustified. Anyway, the problem is that standard
+Horn clauses don't allow universal quantification (forall) or
+implication (if) in goals (though many Prolog engines do support
+them, as an extension). For this reason, we need to accept something
+called "first-order hereditary harrop" (FOHH) clauses – this long
+name basically means "standard Horn clauses with forall and if in
+the body". But it's nice to know the proper name, because there is a
+lot of work describing how to efficiently handle FOHH clauses; see for
+example Gopalan Nadathur's excellent
+"A Proof Procedure for the Logic of Hereditary Harrop Formulas"
+in the bibliography of Chalk Book.
+It turns out that supporting FOHH is not really all that hard. And
+once we are able to do that, we can easily describe the type-checking
+rule for generic functions like foo in our logic.
+Moreover, flattening a bit the definition of clauses given previously, one can
+see that clauses are always of the form:
+hence domain goals are in fact clauses' LHS. That is, at the most granular level,
+domain goals are what the trait solver will end up trying to prove.
+To define the set of domain goals in our system, we need to first
+introduce a few simple formulations. A trait reference consists of
+the name of a trait along with a suitable set of inputs P0..Pn:
+Let's break down each one of these, one-by-one.
+e.g. Implemented(i32: Copy)
+True if the given trait is implemented for the given input types and lifetimes.
+e.g. ProjectionEq<T as Iterator>::Item = u8
+e.g. ProjectionEq<T as Iterator>::Item -> u8
+e.g. FromEnv(Self: Add<i32>)
+e.g. FromEnv(HashSet<K>)
+e.g. Outlives(&'a str: 'b), Outlives('a: 'static)
+True if the given type or region on the left outlives the right-hand region.
+Most goals in our system are "inductive". In an inductive goal,
+circular reasoning is disallowed. Consider this example clause:
+In general, co-inductive traits are used in Rust trait solving when we
+want to enumerate a fixed set of possibilities. In the case of auto
+traits, we are enumerating the set of reachable types from a given
+starting point (i.e., Foo can reach values of type
+Option<Box<Foo>>, which implies it can reach values of type
+Box<Foo>, and then of type Foo, and then the cycle is complete).
+This section covers queries at a fairly high level of abstraction. The
+subsections look a bit more closely at how these ideas are implemented
+in rustc.
+In a traditional Prolog system, when you start a query, the solver
+will run off and start supplying you with every possible answer it can
+find. So given something like this:
+Naturally, in some cases, there may be no possible answers, and hence
+the solver will just give me back no right away:
+In some cases, there might be an infinite number of responses. So for
+example if I gave this query, and I kept hitting y, then the solver
+would never stop giving me back answers:
+Another interesting thing is that queries might still have variables
+in them. For example:
+The trait queries in rustc work somewhat differently. Instead of
+trying to enumerate all possible answers for you, they are looking
+for an unambiguous answer. In particular, when they tell you the
+value for a type variable, that means that this is the only possible
+instantiation that you could use, given the current set of impls and
+where-clauses, that would be provable.
+Therefore, the result we get back would be as follows (I'm going to
+ignore region constraints and the "value"):
+In short, the query result says that it is too soon to say much about
+whether this trait is proven. During type-checking, this is not an
+immediate error: instead, the type checker would hold on to this
+requirement (Vec<?T>: Borrow<?U>) and wait. As we'll see in the next
+example, it may happen that ?T and ?U wind up constrained from
+other sources, in which case we can try the trait query again.
+Let's suppose that the type checker decides to revisit the
+"as-yet-unproven" trait obligation we saw before, Vec<?T>: Borrow<?U>. ?U is no longer an unbound inference variable; it now
+has a value, Vec<?V>. So, if we "refresh" the query with that value, we get:
+Here, it is saying that we have indeed proven that the obligation
+holds, and we also know that ?T and ?V are the same type (but we
+don't know what that type is yet!).
+(In fact, as the function ends here, the type checker would give an
+error at this point, since the element types of t and u are still
+not yet known, even though they are known to be the same.)
+This work is ongoing. This section will be filled in once some of it has landed in rustc.
+Method lookup can be rather complex due to the interaction of a number
+of factors, such as self types, autoderef, trait lookup, etc. This
+file provides an overview of the process. More detailed notes are in
+the code itself, naturally.
+One reason for this division is to be more amenable to caching. The
+probe phase produces a "pick" (probe::Pick), which is designed to be
+cacheable across method-call sites. Therefore, it does not include
+inference variables or other information.
+Candidates are grouped into two kinds, inherent and extension.
+FIXME: Inherent candidates are not always derived from impls. If you
+have a trait object, such as a value of type Box<ToString>, then the
+trait methods (to_string(), in this case) are inherently associated
+with it. Another case is type parameters, in which case the methods of
+their bounds are inherent. However, this part of the rules is subject
+to change: when DST's "impl Trait for Trait" is complete, trait object
+dispatch could be subsumed into trait matching, and the type parameter
+behavior should be reconsidered in light of where clauses.
+So, let's continue our example. Imagine that we were calling a method
+foo with the receiver Rc<Box<[T; 3]>> and there is a trait Foo
+that defines it with &self for the type Rc<U> as well as a method
+on the type Box that defines Foo but with &mut self. Then we
+might have two candidates:
+Finally, to actually pick the method, we will search down the steps,
+trying to match the receiver type against the candidate types. At
+each step, we also consider an auto-ref and auto-mut-ref to see whether
+that makes any of the candidates match. We pick the first step where
+we find a match.
+During type checking we must infer the variance of type and lifetime
+parameters. The algorithm is taken from Section 4 of the paper "Taming the
+Wildcards: Combining Definition- and Use-Site Variance" published in
+PLDI'11 and written by Altidor et al., and hereafter referred to as The Paper.
+We do not infer variance for type parameters found on traits, functions,
+or impls. Variance on trait parameters can indeed make sense
+(and we used to compute it) but it is actually rather subtle in
+meaning and not that useful in practice, so we removed it. See the
+addendum for some details. Variances on function/impl parameters, on the
+other hand, doesn't make sense because these parameters are instantiated and
+then forgotten, they don't persist in types or compiled byproducts.
+The basic idea is quite straightforward. We iterate over the types
+defined and, for each use of a type parameter X, accumulate a
+constraint indicating that the variance of X must be valid for the
+variance of that use site. We then iteratively refine the variance of
+X until all constraints are met. There is always a solution, because at
+the limit we can declare all type parameters to be invariant and all
+constraints will be satisfied.
+These indicate that (1) the variance of A must be at most covariant;
+(2) the variance of B must be at most contravariant; and (3, 4) the
+variance of C must be at most covariant and contravariant. All of these
+results are based on a variance lattice defined as follows:
+You may be wondering why fixed-point iteration is required. The reason
+is that the variance of a use site may itself be a function of the
+variance of other type parameters. In full generality, our constraints
+take the form:
+Because variance is a whole-crate inference, its dependency graph
+can become quite muddled if we are not careful. To resolve this, we refactor
+into two queries:
+As mentioned above, we used to permit variance on traits. This was
+computed based on the appearance of trait type parameters in
+method signatures and was used to represent the compatibility of
+vtables in trait objects (and also "virtual" vtables or dictionary
+in trait bounds). One complication was that variance for
+associated types is less obvious, since they can be projected out
+and put to myriad uses, so it's not clear when it is safe to allow
+X<A>::Bar to vary (or indeed just what that means). Moreover (as
+covered below) all inputs on any trait with an associated type had
+to be invariant, limiting the applicability. Finally, the
+annotations (MarkerTrait, PhantomFn) needed to ensure that all
+trait type parameters had a variance were confusing and annoying
+for little benefit.
+Just for historical reference, I am going to preserve some text indicating how
+one could interpret variance and trait matching.
+Just as with structs and enums, we can decide the subtyping
+relationship between two object types &Trait<A> and &Trait<B>
+based on the relationship of A and B. Note that for object
+types we ignore the Self type parameter – it is unknown, and
+the nature of dynamic dispatch ensures that we will always call a
+function that is expected the appropriate Self type. However, we
+must be careful with the other type parameters, or else we could
+end up calling a function that is expecting one type but provided
+another.
+But traits aren't only used with objects. They're also used when
+deciding whether a given impl satisfies a given trait bound. To set the
+scene here, imagine I had a function:
+OK, so intuitively we want this to be legal, so let's bring this back
+to variance and see whether we are computing the correct result. We
+must first figure out how to phrase the question "is an impl for
+Object,i32 usable where an impl for String,i32 is expected?"
+Maybe it's helpful to think of a dictionary-passing implementation of
+type classes. In that case, convertAll() takes an implicit parameter
+representing the impl. In short, we have an impl of type:
+These conditions are satisfied and so we are happy.
+Traits with associated types – or at minimum projection
+expressions – must be invariant with respect to all of their
+inputs. To see why this makes sense, consider what subtyping for a
+trait reference means:
+Another related reason is that if we didn't make traits with
+associated types invariant, then projection is no longer a
+function with a single result. Consider:
+Opaque types are syntax to declare an opaque type alias that only
+exposes a specific set of traits as their interface; the concrete type in the
+background is inferred from a certain set of use sites of the opaque type.
+Since there needs to be a concrete background type, you can currently
+express that type by using the opaque type in a "defining use site".
+Any other "defining use site" needs to produce the exact same type.
+Currently only the return value of a function can be a defining use site
+of an opaque type (and only if the return type of that function contains
+the opaque type).
+Associated opaque types can be defined by any other associated item
+on the same trait impl or a child of these associated items. For instance:
+In Rust, pattern matching and bindings have a few very helpful properties. The
+compiler will check that bindings are irrefutable when made and that match arms
+are exhaustive.
+The borrow check is Rust's "secret sauce" – it is tasked with
+enforcing a number of properties:
+The borrow checker operates on the MIR. An older implementation operated on the
+HIR. Doing borrow checking on MIR has several advantages:
+Part of the borrow checker's job is to track which variables are
+"initialized" at any given point in time -- this also requires
+figuring out where moves occur and tracking those.
+From a user's perspective, initialization -- giving a variable some
+value -- and moves -- transferring ownership to another place -- might
+seem like distinct topics. Indeed, our borrow checker error messages
+often talk about them differently. But within the borrow checker,
+they are not nearly as separate. Roughly speaking, the borrow checker
+tracks the set of "initialized places" at any point in the source
+code. Assigning to a previously uninitialized local variable adds it
+to that set; moving from a local variable removes it from that set.
+To make it easier to peruse, this section is broken into a number of
+subsections:
+In reality, it's not enough to track initialization at the granularity
+of local variables. Rust also allows us to do moves and initialization
+at the field granularity:
+One of the first things we do in the MIR borrow check is to construct
+the set of move paths. This is done as part of the
+MoveData::gather_moves function. This function uses a MIR visitor
+called Gatherer to walk the MIR and look at how each Place
+within is accessed. For each such Place, it constructs a
+corresponding MovePathIndex. It also records when/where that
+particular move path is moved/initialized, but we'll get to that in a
+later section.
+As we noted above, move-paths are stored in a big vector and
+referenced via their MovePathIndex. However, within this vector,
+they are also structured into a tree. So for example if you have the
+MovePathIndex for a.b.c, you can go to its parent move-path
+a.b. You can also iterate over all children paths: so, from a.b,
+you might iterate to find the path a.b.c (here you are iterating
+just over the paths that are actually referenced in the source,
+not all possible paths that could have been referenced). These
+references are used for example in the
+find_in_move_path_or_its_descendants function, which determines
+whether a move-path (e.g., a.b) or any child of that move-path
+(e.g.,a.b.c) matches a given predicate.
+Before we can infer the value of regions, we need to collect
+constraints on the regions. The full set of constraints is described
+in the section on constraint propagation, but the two most
+common sorts of constraints are:
+Here is the high-level idea: we start off each region with the MIR locations we
+know must be in it from the liveness constraints. From there, we use all of the
+outlives constraints computed from the type checker to propagate the
+constraints: for each region 'a, if 'a: 'b, then we add all elements of
+'b to 'a, including end('b). This all happens in
+propagate_constraints.
+Then, we will check for errors. We first check that type tests are satisfied by
+calling check_type_tests. This checks constraints like T: 'a. Second, we
+check that universal regions are not "too big". This is done by calling
+check_universal_regions. This checks that for each region 'a if 'a
+contains the element end('b), then we must already know that 'a: 'b holds
+(e.g. from a where clause). If we don't already know this, that is an error...
+well, almost. There is some special handling for closures that we will discuss
+later.
+Let's back up a bit. We need to introduce some free inference variables (as is
+done in replace_regions_in_mir). This example doesn't use the exact regions
+produced, but it (hopefully) is enough to get the idea across.
+Now we use the outlives constraints to expand each region. Specifically, we
+know that '#2: '#3 ...
+... and '#1: '#2, so ...
+Now, we need to check that no regions were too big (we don't have any type
+tests to check in this case). Notice that '#1 now contains end('#3), but
+we have no where clause or implied bound to say that 'a: 'b... that's an
+error!
+In this chapter, we'll explain the "heart" of constraint propagation,
+covering both liveness and outlives constraints.
+Conceptually, region inference is a "fixed-point" computation. It is
+given some set of constraints {C} and it computes a set of values
+Values: R -> {E} that maps each region R to a set of elements
+{E} (see here for more notes on region elements):
+In practice, however, we are a bit more clever. Instead of applying
+the constraints in a loop, we can analyze the constraints and figure
+out the correct order to apply them, so that we only have to apply
+each constraint once in order to find the final result.
+The liveness values are computed in the type-check and passed to the
+region inference upon creation in the liveness_constraints argument.
+These are not represented as individual constraints like R live at E
+though; instead, we store a (sparse) bitset per region variable (of
+type LivenessValues). This way we only need a single bit for each
+liveness constraint.
+One thing that is worth mentioning: All lifetime parameters are always
+considered to be live over the entire function body. This is because
+they correspond to some portion of the caller's execution, and that
+execution clearly includes the time spent in this function, since the
+caller is waiting for us to return.
+In the code, the set of outlives constraints is given to the region
+inference context on creation in a parameter of type
+OutlivesConstraintSet. The constraint set is basically just a list of 'a: 'b constraints.
+When using a graph representation, we can detect regions that must be equal
+by looking for cycles. That is, if you have a constraint like
+Therefore, one of the first things that we do in propagating region
+values is to compute the strongly connected components (SCCs) in
+the constraint graph. The result is stored in the constraint_sccs
+field. You can then easily find the SCC that a region r is a part of
+by invoking constraint_sccs.scc(r).
+Working in terms of SCCs allows us to be more efficient: if we have a
+set of regions 'a...'d that are part of a single SCC, we don't have
+to compute/store their values separately. We can just store one value
+for the SCC, since they must all be equal.
+If you look over the region inference code, you will see that a number
+of fields are defined in terms of SCCs. For example, the
+scc_values field stores the values of each SCC. To get the value
+of a specific region 'a then, we first figure out the SCC that the
+region is a part of, and then find the value of that SCC.
+When we compute SCCs, we not only figure out which regions are a
+member of each SCC, we also figure out the edges between them. So for example
+consider this set of outlives constraints:
+The liveness constraints that come in from the type-checker are
+expressed in terms of regions -- that is, we have a map like
+Liveness: R -> {E}. But we want our final result to be expressed
+in terms of SCCs -- we can integrate these liveness constraints very
+easily just by taking the union:
+Once we have computed the DAG of SCCs, we use that to structure out
+entire computation. If we have an edge S1 -> S2 between two SCCs,
+that means that Values(S1) >= Values(S2) must hold. So, to compute
+the value of S1, we first compute the values of each successor S2.
+Then we simply union all of those values together. To use a
+quasi-iterator-like notation:
+"Universal regions" is the name that the code uses to refer to "named
+lifetimes" -- e.g., lifetime parameters and 'static. The name
+derives from the fact that such lifetimes are "universally quantified"
+(i.e., we must make sure the code is true for all values of those
+lifetimes). It is worth spending a bit of discussing how lifetime
+parameters are handled during region inference. Consider this example:
+In fact, the universal regions can be further subdivided based on
+where they were brought into scope (see the RegionClassification
+type). These subdivisions are not important for the topics discussed
+here, but become important when we consider closure constraint
+propagation, so we discuss them there.
+During region inference, we compute a value for each universal region
+in the same way as we compute values for other regions. This value
+represents, effectively, the lower bound on that universal region
+-- the things that it must outlive. We now describe how we use this
+value to check for errors.
+All universal regions have an initial liveness constraint that
+includes the entire function body. This is because lifetime parameters
+are defined in the caller and must include the entirety of the
+function call that invokes this particular function. In addition, each
+universal region 'a includes itself (that is, end('a)) in its
+liveness constraint (i.e., 'a must extend until the end of
+itself). In the code, these liveness constraints are setup in
+init_free_and_bound_regions.
+Once we have finished constraint propagation, we then enforce a
+constraint that if some universal region 'a includes an element
+end('b), then 'a: 'b must be declared in the function's bounds. If
+not, as in our example, that is an error. This check is done in the
+check_universal_regions function, which simply iterates over all
+universal regions, inspects their final value, and tests against the
+declared UniversalRegionRelations.
+Here, the true return type (often called the "hidden type") is only
+permitted to capture the lifetimes 'a or 'b. You can kind of see
+this more clearly by desugaring that impl Trait return type into its
+more explicit form:
+Here, the idea is that the hidden type must be some type that could
+have been written in place of the impl Trait<'x, 'y> -- but clearly
+such a type can only reference the regions 'x or 'y (or
+'static!), as those are the only names in scope. This limitation is
+then translated into a restriction to only access 'a or 'b because
+we are returning MakeReturn<'a, 'b>, where 'x and 'y have been
+replaced with 'a and 'b respectively.
+At present, the "choice" regions from a member constraint are always
+lifetime parameters from the current function. This falls out from the
+placement of impl Trait, though in the future it may not be the case.
+We take some advantage of this fact, as it simplifies the current
+code. In particular, we don't have to consider a case like '0 member of ['1, 'static], in which the value of both '0 and '1 are being
+inferred and hence changing. See rust-lang/rust#61773 for more
+information.
+Member constraints are a bit more complex than other forms of
+constraints. This is because they have a "or" quality to them -- that
+is, they describe multiple choices that we must select from. E.g., in
+our example constraint '0 member of ['a, 'b, 'static], it might be
+that '0 is equal to 'a, 'b, or 'static. How can we pick the
+correct one? What we currently do is to look for a minimal choice
+-- if we find one, then we will grow '0 to be equal to that minimal
+choice. To find that minimal choice, we take two factors into
+consideration: lower and upper bounds.
+Unfortunately, in our example, this is not very helpful. The lower
+bound for '0 will just be the liveness set {L}, and we know that
+all the lifetime parameters outlive that set. So we are left with the
+same set of choices here. (But in other examples, particularly those
+with different variance, lower bound constraints may be relevant.)
+We can use upper bounds to rule out members in a very similar way to
+lower lower bounds. If UB is some upper bound, then we know that UB: '0 must hold, so we can rule out any choice 'choice where UB: 'choice does not hold.
+After applying lower and upper bounds, we can still sometimes have
+multiple possibilities. For example, imagine a variant of our example
+using types with the opposite variance. In that case, we would have
+the constraint '0: 'a instead of 'a: '0. Hence the current value
+of '0 would be {L, 'a}. Using this as a lower bound, we would be
+able to narrow down the member choices to ['a, 'static] because 'b: 'a is not known to hold (but 'a: 'a and 'static: 'a do hold). We
+would not have any upper bounds, so that would be our final set of choices.
+This choice is consistent with the general 'flow' of region
+propagation, which always aims to compute a minimal value for the
+region being inferred. However, it is somewhat arbitrary.
+In practice, computing upper bounds is a bit inconvenient, because our
+data structures are setup for the opposite. What we do is to compute
+the reverse SCC graph (we do this lazily and cache the result) --
+that is, a graph where 'a: 'b induces an edge SCC('b) -> SCC('a). Like the normal SCC graph, this is a DAG. We can then do a
+depth-first search starting from SCC('0) in this graph. This will
+take us to all the SCCs that must outlive '0.
+One wrinkle is that, as we walk the "upper bound" SCCs, their values
+will not yet have been fully computed. However, we have already
+applied their liveness constraints, so we have some information about
+their value. In particular, for any regions representing lifetime
+parameters, their value will contain themselves (i.e., the initial
+value for 'a includes 'a and the value for 'b contains 'b). So
+we can collect all of the lifetime parameters that are reachable,
+which is precisely what we are interested in.
+From time to time we have to reason about regions that we can't
+concretely know. For example, consider this program:
+We handle this sort of subtyping by taking the variables that are
+bound in the supertype and replacing them with
+universally quantified
+representatives, denoted like !1 here. We call these regions "placeholder
+regions" – they represent, basically, "some unknown region".
+So let's work through what happens next. To check if two functions are
+subtypes, we check if their arguments have the desired relationship
+(fn arguments are contravariant, so
+we swap the left and right here):
+In the previous section, we introduced the idea of a placeholder
+region, and we denoted it !1. We call this number 1 the universe
+index. The idea of a "universe" is that it is a set of names that
+are in scope within some type or at some point. Universes are formed
+into a tree, where each child extends its parents with some new names.
+So the root universe conceptually contains global names, such as
+the the lifetime 'static or the type i32. In the compiler, we also
+put generic type parameters into this root universe (in this sense,
+there is not just one root universe, but one per item). So consider
+this function bar:
+The idea is that this child universe U1 extends the root universe U0
+with a new name, which we are identifying by its universe number:
+!1.
+This implies that, while in U2, we can name things from U0 or U2, but
+not U1.
+How can we get away with this? Doesn't this mean that we would allow
+U2 to also see U1? The answer is that, yes, we would, if that
+question ever arose. But because of the structure of our type
+checker etc, there is no way for that to happen. In order for
+something happening in the universe U1 to "communicate" with something
+happening in U2, they would have to have a shared inference variable X
+in common. And because everything in U1 is scoped to just U1 and its
+children, that inference variable X would have to be in U0. And since
+X is in U0, it cannot name anything from U1 (or U2). This is perhaps easiest
+to see by using a kind of generic "logic" example:
+Here, the only way for the two foralls to interact would be through X,
+but neither Y nor Z are in scope when X is declared, so its value
+cannot reference either of them.
+But where does that error come from? The way it happens is like this.
+When we are constructing the region inference context, we can tell
+from the type inference context how many placeholder variables exist
+(the InferCtxt has an internal counter). For each of those, we
+create a corresponding universal region variable !n and a "region
+element" placeholder(n). This corresponds to "some unknown set of other
+elements". The value of !n is {placeholder(n)}.
+When we encounter this constraint, the ordinary procedure is to start
+a DFS from P. We keep walking so long as the nodes we are walking
+are present in value(V2) and we add those nodes to value(V1). If
+we reach a return point, we add in any end(X) elements. That part
+remains unchanged.
+Now there are two ways that could happen. First, if U(V1) can see
+the universe x (i.e., x <= U(V1)), then we can just add placeholder(x)
+to value(V1) and be done. But if not, then we have to approximate:
+we may not know what set of elements placeholder(x) represents, but we
+should be able to compute some sort of upper bound B for it –
+some region B that outlives placeholder(x). For now, we'll just use
+'static for that (since it outlives everything) – in the future, we
+can sometimes be smarter here (and in fact we have code for doing this
+already in other contexts). Moreover, since 'static is in the root
+universe U0, we know that all variables can see it – so basically if
+we find that value(V2) contains placeholder(x) for some universe x
+that V1 can't see, then we force V1 to 'static.
+After all constraints have been propagated, the NLL region inference
+has one final check, where it goes over the values that wound up being
+computed for each universal region and checks that they did not get
+'too large'. In our case, we will go through each placeholder region
+and check that it contains only the placeholder(u) element it is known to
+outlive. (Later, we might be able to know that there are relationships
+between two placeholder regions and take those into account, as we do
+for universal regions from the fn signature.)
+Put another way, the "universal regions" check can be considered to be
+checking constraints like:
+OK, so far so good. Now let's walk through what would happen with our
+first example:
+At that point, constraint propagation is complete, because all the
+outlives relationships are satisfied. Then we would go to the "check
+universal regions" portion of the code, which would test that no
+universal region grew too large.
+Here we would replace the bound region in the supertype with a placeholder, as before, yielding:
+then we instantiate the variable on the left-hand side with an
+existential in universe U2, yielding the following (?n is a notation
+for an existential variable):
+(This should surprise you a little. It surprised me when I first realized it.
+We are saying that if we are a fn that needs both of its arguments to have
+the same region, we can accept being called with arguments with two
+distinct regions. That seems intuitively unsound. But in fact, it's fine, as
+I tried to explain in this issue on the Rust issue
+tracker long ago. The reason is that even if we get called with arguments of
+two distinct lifetimes, those two lifetimes have some intersection (the call
+itself), and that intersection can be our value of 'a that we use as the
+common lifetime of our arguments. -nmatsakis)
+Let's look at one last example. We'll extend the previous one to have
+a return type:
+Despite seeming very similar to the previous example, this case is going to get
+an error. That's good: the problem is that we've gone from a fn that promises
+to return one of its two arguments, to a fn that is promising to return the
+first one. That is unsound. Let's see how it plays out.
+And finally the outlives relationships. Here, let V1, V2, and V3 be the
+variables we assign to !1, !2, and ?3 respectively:
+Now constraint propagation is done, but when we check the outlives
+relationships, we find that V2 includes this new element placeholder(1),
+so we report an error.
+When we are checking the type tests and universal regions, we may come
+across a constraint that we can't prove yet if we are in a closure
+body! However, the necessary constraints may actually hold (we just
+don't know it yet). Thus, if we are inside a closure, we just collect
+all the constraints we can't prove yet and return them. Later, when we
+are borrow check the MIR node that created the closure, we can also
+check that these constraints hold. At that time, if we can't prove
+they hold, we report an error.
+TODO: we should discuss how to generate errors from the results of these analyses.
+Two-phase borrows are a more permissive version of mutable borrows that allow
+nested method calls such as vec.push(vec.len()). Such borrows first act as
+shared borrows in a "reservation" phase and can later be "activated" into a
+full mutable borrow.
+Each two-phase borrow is assigned to a temporary that is only used once. As
+such we can define:
+Two-phase borrows are treated as if they were mutable borrows with the
+following exceptions:
+When working with associated and/or or generic items (types, constants,
+functions/methods) it is often relevant to have more information about the
+Self or generic parameters. Trait bounds and similar information is encoded in
+the ParamEnv. Often this is not enough information to obtain things like the
+type's Layout, but you can do all kinds of other checks on it (e.g. whether a
+type implements Copy) or you can evaluate an associated constant whose value
+does not depend on anything from the parameter environment.
+All of the preceding chapters of this guide have one thing in common: we never
+generated any executable machine code at all! With this chapter, all of that
+changes.
+So far, we've shown how the compiler can take raw source code in text format
+and transform it into MIR. We have also shown how the compiler does various
+analyses on the code to detect things like type or lifetime errors. Now, we
+will finally take the MIR and produce some executable machine code.
+If you'd like a very high-level introduction to MIR, as well as some
+of the compiler concepts that it relies on (such as control-flow
+graphs and desugaring), you may enjoy the
+rust-lang blog post that introduced MIR.
+You can get a feeling for how MIR is structed by translating simple
+programs into MIR and reading the pretty printed output. In fact, the
+playground makes this easy, since it supplies a MIR button that will
+show you the MIR for your program. Try putting this program into play
+(or clicking on this link), and then clicking the "MIR"
+button on the top:
+Terminators are different from statements because they can have more
+than one successor – that is, control may flow to different
+places. Function calls like the call to Vec::new are always
+terminators because of the possibility of unwinding, although in the
+case of Vec::new we are able to see that indeed unwinding is not
+possible, and hence we list only one successor block, bb2.
+As you can see from this grammar, rvalues cannot be nested – they can
+only reference places and constants. Moreover, when you use a place,
+we indicate whether we are copying it (which requires that the
+place have a type T where T: Copy) or moving it (which works
+for a place of any type). So, for example, if we had the expression x = a + b + c in Rust, that would get compiled to two statements and a
+temporary:
+MIR optimizations run after borrow checking. We run a series of optimization
+passes over the MIR to improve it. Some passes are required to run on all code,
+some passes don't actually do optimizations but only check stuff, and some
+passes are only turned on in release mode.
+You can also make more selective filters. For example, main & CleanEndRegions
+will select for things that reference both main and the pass
+CleanEndRegions:
+Constant evaluation is the process of computing values at compile time. For a
+specific item (constant/static/array length) this happens after the MIR for the
+item is borrow-checked and optimized. In many cases trying to const evaluate an
+item will trigger the computation of its MIR for the first time.
+Additionally constant evaluation can be used to reduce the workload or binary
+size at runtime by precomputing complex operations at compiletime and only
+storing the result.
+rustc doesn't actually invoke anything until the constant is either used or
+placed into metadata.
+The compiler needs to figure out the length of the array before being able to
+create items that use the type (locals, constants, function arguments, ...).
+Future evaluations of the same constants will not actually invoke
+Miri, but just use the cached result.
+To support any kind of pointers, Miri needs to have a "virtual memory" that the
+pointers can point to. This is implemented in the Memory type. In the
+simplest model, every global variable, stack variable and every dynamic
+allocation corresponds to an Allocation in that memory. (Actually using an
+allocation for every MIR stack variable would be very inefficient; that's why we
+have Operand::Immediate for stack variables that are both small and never have
+their address taken. But that is purely an optimization.)
+These allocations exist so that references and raw pointers have something to
+point to. There is no global linear heap in which things are allocated, but each
+allocation (be it for a local variable, a static or a (future) heap allocation)
+gets its own little memory with exactly the required size. So if you have a
+pointer to an allocation for a local variable a, there is no possible (no
+matter how unsafe) operation that you can do that would ever change said pointer
+to a pointer to a different local variable b.
+Pointer arithmetic on a will only ever change its offset; the AllocId stays the same.
+Miri starts by creating a virtual stack frame for the current constant that is
+being evaluated. There's essentially no difference between a constant and a
+function with no arguments, except that constants do not allow local (named)
+variables at the time of writing this guide.
+As you probably know, rust has a very expressive type system that has extensive
+support for generic types. But of course, assembly is not generic, so we need
+to figure out the concrete types of all the generics before the code can
+execute.
+Different languages handle this problem differently. For example, in some
+languages, such as Java, we may not know the most precise type of value until
+runtime. In the case of Java, this is ok because (almost) all variables are
+reference values anyway (i.e. pointers to a stack allocated object). This
+flexibility comes at the cost of performance, since all accesses to an object
+must dereference a pointer.
+Monomorphization is the first step in the backend of the rust compiler.
+First, we need to figure out what concrete types we need for all the generic
+things in our program. This is called collection, and the code that does this
+is called the monomorphization collector.
+As mentioned above, monomorphization produces fast code, but it comes at the
+cost of compile time and binary size. MIR
+optimizations can help a bit with this. Another
+optimization currently under development is called polymorphization.
+The general idea is that often we can share some code between monomorphized
+copies of code. More precisely, if a MIR block is not dependent on a type
+parameter, it may not need to be monomorphized into many copies. Consider the
+following example:
+Now that we have a list of symbols to generate from the collector, we need to
+generate some sort of codegen IR. In this chapter, we will assume LLVM IR,
+since that's what rustc usually uses. The actual monomorphization is performed
+as we go, while we do the translation.
+Before a function is translated a number of simple and primitive analysis
+passes will run to help us generate simpler and more efficient LLVM IR. An
+example of such an analysis pass would be figuring out which variables are
+SSA-like, so that we can translate them to SSA directly rather than relying on
+LLVM's mem2reg for those variables. The analysis can be found in
+rustc_codegen_ssa::mir::analyze.
+Usually a single MIR basic block will map to a LLVM basic block, with very few
+exceptions: intrinsic or function calls and less basic MIR statements like
+assert can result in multiple basic blocks. This is a perfect lede into the
+non-portable LLVM-specific part of the code generation. Intrinsic generation is
+fairly easy to understand as it involves very few abstraction levels in between
+and can be found in rustc_codegen_llvm::intrinsic.
+Code generation or "codegen" is the part of the compiler that actually
+generates an executable binary. Usually, rustc uses LLVM for code generation;
+there is also support for Cranelift. The key is that rustc doesn't implement
+codegen itself. It's worth noting, though, that in the rust source code, many
+parts of the backend have codegen in their names (there are no hard
+boundaries).
+LLVM takes input in the form of LLVM IR. It is basically assembly code with
+additional low-level types and annotations added. These annotations are helpful
+for doing optimizations on the LLVM IR and outputted machine code. The end
+result of all this is (at long last) something executable (e.g. an ELF object,
+an EXE, or wasm).
+Once LLVM IR for all of the functions and statics, etc is built, it is time to
+start running LLVM and its optimization passes. LLVM IR is grouped into
+"modules". Multiple "modules" can be codegened at the same time to aid in
+multi-core utilization. These "modules" are what we refer to as codegen
+units. These units were established way back during monomorphization
+collection phase.
+Once LLVM produces objects from these modules, these objects are passed to the
+linker along with, optionally, the metadata object and an archive or an
+executable is produced.
+It is not necessarily the codegen phase described above that runs the
+optimizations. With certain kinds of LTO, the optimization might happen at the
+linking time instead. It is also possible for some optimizations to happen
+before objects are passed on to the linker and some to happen during the
+linking.
+This all happens towards the very end of compilation. The code for this can be
+found in librustc_codegen_ssa::back and
+librustc_codegen_llvm::back. Sadly, this piece of code is not
+really well-separated into LLVM-dependent code; the rustc_codegen_ssa
+contains a fair amount of code specific to the LLVM backend.
+Once these components are done with their work you end up with a number of
+files in your filesystem corresponding to the outputs you have requested.
+The Rust compiler uses LLVM as its primary codegen backend today, and naturally
+we want to at least occasionally update this dependency! Currently we do not
+have a strict policy about when to update LLVM or what it can be updated to, but
+a few guidelines are applied:
+This policy may change over time (or may actually start to exist as a formal
+policy!), but for now these are rough guidelines!
+There are a few reasons nowadays that we want to update LLVM in one way or
+another:
+Each of these reasons has a different strategy for updating LLVM, and we'll go
+over them in detail here.
+For updates of LLVM that are to fix a small bug, we cherry-pick the bugfix to
+the branch we're already using. The steps for this are:
+The tl;dr; is that we can cherry-pick bugfixes at any time and pull them back
+into the rust-lang/llvm-project branch that we're using, and getting it into the
+compiler is just updating the submodule via a PR!
+Unlike bugfixes, updating to pick up a new feature of LLVM typically requires a
+lot more work. This is where we can't reasonably cherry-pick commits backwards
+so we need to do a full update. There's a lot of stuff to do here, so let's go
+through each in detail.
+Ideally the above instructions are pretty smooth, but here's some caveats to
+keep in mind while going through them:
+Updating to a new release of LLVM is very similar to the "feature updates"
+section above. The release process for LLVM is often months-long though and we
+like to ensure compatibility ASAP. The main tweaks to the "feature updates"
+section above is generally around branch naming. The sequence of events
+typically looks like:
+This section is about debugging compiler bugs in code generation (e.g. why the
+compiler generated some piece of code or crashed in LLVM). LLVM is a big
+project on its own that probably needs to have its own debugging document (not
+that I could find one). But here are some tips that are important in a rustc
+context:
+As a general rule, compilers generate lots of information from analyzing code.
+Thus, a useful first step is usually to find a minimal example. One way to do
+this is to
+The official compilers (including nightlies) have LLVM assertions disabled,
+which means that LLVM assertion failures can show up as compiler crashes (not
+ICEs but "real" crashes) and other sorts of weird behavior. If you are
+encountering these, it is a good idea to try using a compiler with LLVM
+assertions enabled - either an "alt" nightly or a compiler you build yourself
+by setting [llvm] assertions=true in your config.toml - and see whether
+anything turns up.
+The default rustc compilation pipeline has multiple codegen units, which is
+hard to replicate manually and means that LLVM is called multiple times in
+parallel. If you can get away with it (i.e. if it doesn't make your bug
+disappear), passing -C codegen-units=1 to rustc will make debugging easier.
+If you just want to get the LLVM IR during the LLVM pipeline, to e.g. see which
+IR causes an optimization-time assertion to fail, or to see when LLVM performs
+a particular optimization, you can pass the rustc flag -C llvm-args=-print-after-all, and possibly add -C llvm-args='-filter-print-funcs=EXACT_FUNCTION_NAME (e.g. -C llvm-args='-filter-print-funcs=_ZN11collections3str21_$LT$impl$u20$str$GT$\ 7replace17hbe10ea2e7c809b0bE').
+That produces a lot of output into standard error, so you'll want to pipe that
+to some file. Also, if you are using neither -filter-print-funcs nor -C codegen-units=1, then, because the multiple codegen units run in parallel, the
+printouts will mix together and you won't be able to read anything.
+If you want just the IR for a specific function (say, you want to see why it
+causes an assertion or doesn't optimize correctly), you can use llvm-extract,
+e.g.
+When filing an LLVM bug report, you will probably want some sort of minimal
+working example that demonstrates the problem. The Godbolt compiler explorer is
+really helpful for this.
+Once you've identified the bug as an LLVM bug, you will sometimes
+find that it has already been reported and fixed in LLVM, but we haven't
+gotten the fix yet (or perhaps you are familiar enough with LLVM to fix it yourself).
+In that case, we can sometimes opt to port the fix for the bug
+directly to our own LLVM fork, so that rustc can use it more easily.
+Our fork of LLVM is maintained in rust-lang/llvm-project. Once
+you've landed the fix there, you'll also need to land a PR modifying
+our submodule commits -- ask around on Zulip for help.
+All the code related to the compilation of MIR into LLVM IR was contained
+inside the rustc_codegen_llvm crate. Here is the breakdown of the most
+important elements:
+The goal of this refactoring is to separate inside this crate code that is
+specific to the LLVM from code that can be reused for other rustc backends. For
+instance, the mir folder is almost entirely backend-specific but it relies
+heavily on other parts of the crate. The separation of the code must not affect
+the logic of the code nor its performance.
+For these reasons, the separation process involves two transformations that
+have to be done at the same time for the resulting code to compile :
+Although there are already many lifetime parameters in the code, making it
+generic uncovered situations where the borrow-checker was passing only due to
+the special nature of the LLVM objects manipulated (they are extern pointers).
+For instance, an additional lifetime parameter had to be added to
+LocalAnalyser in analyse.rs, leading to the definition:
+In this signature, we have the two lifetime parameters explained earlier and
+the master type Bx which satisfies the trait BuilderMethods corresponding
+to the interface satisfied by the Builder struct. The BuilderMethods
+defines an associated type Bx::CodegenCx that itself satisfies the
+CodegenMethods traits implemented by the struct CodegenCx.
+During the traitification process, certain functions have been converted from
+methods of a local structure to methods of CodegenCx or Builder and a
+corresponding self parameter has been added. Indeed, LLVM stores information
+internally that it can access when called through its API. This information
+does not show up in a Rust data structure carried around when these methods are
+called. However, when implementing a Rust backend for rustc, these methods
+will need information from CodegenCx, hence the additional parameter (unused
+in the LLVM implementation of the trait).
+The traits offer an API which is very similar to the API of LLVM. This is not
+the best solution since LLVM has a very special way of doing things: when
+addding another backend, the traits definition might be changed in order to
+offer more flexibility.
+However, the current separation between backend-agnostic and LLVM-specific code
+has allowed the reuse of a significant part of the old rustc_codegen_llvm.
+Here is the new LOC breakdown between backend-agnostic (BA) and LLVM for the
+most important elements:
+There are two main phases to returning the caller location in a const context: walking up the stack
+to find the right location and allocating a const value to return.
+In a const context we "walk up the stack" from where the intrinsic is invoked, stopping when we
+reach the first function call in the stack which does not have the attribute. This walk is in
+InterpCx::find_closest_untracked_caller_location().
+To generate efficient code for a tracked function and its callers, we need to provide the same
+behavior from the intrinsic's point of view without having a stack to walk up at runtime. We invert
+the approach: as we grow the stack down we pass an additional argument to calls of tracked functions
+rather than walking up the stack when the intrinsic is called. That additional argument can be
+returned wherever the caller location is queried.
+When generating a call to a function which is tracked, we pass the location argument the value of
+FunctionCx::get_caller_location.
+We more efficiently achieve the same behavior as a loop starting from the bottom by passing a single
+&Location value through the caller_location fields of multiple FunctionCxs as we grow the
+stack downward.
+What does this transformation look like in practice? Take this example which uses the new feature:
+In codegen contexts we have to modify the callee ABI to pass this information down the stack, but
+the attribute expressly does not modify the type of the function. The ABI change must be
+transparent to type checking and remain sound in all uses.
+Direct calls to tracked functions will always know the full codegen flags for the callee and can
+generate appropriate code. Indirect callers won't have this information and it's not encoded in
+the type of the function pointer they call, so we generate a ReifyShim around the function
+whenever taking a pointer to it. This shim isn't able to report the actual location of the indirect
+call (the function's definition site is reported instead), but it prevents miscompilation and is
+probably the best we can do without modifying fully-stabilized type signatures.
+When applied to trait method implementations, the attribute works as it does for regular functions.
+When applied to a trait method prototype, the attribute applies to all implementations of the
+method. When applied to a default trait method implementation, the attribute takes effect on
+that implementation and any overrides.
+Broadly speaking, this feature's goal is to improve common Rust error messages without breaking
+stability guarantees, requiring modifications to end-user source, relying on platform-specific
+debug-info, or preventing user-defined types from having the same error-reporting benefits.
+Improving the output of these panics has been a goal of proposals since at least mid-2016 (see
+non-viable alternatives in the approved RFC for details). It took two more years until RFC 2091
+was approved, much of its rationale for this feature's design having been discovered through the
+discussion around several earlier proposals.
+The design in the original RFC limited itself to implementations that could be done inside the
+compiler at the time without significant refactoring. However in the year and a half between the
+approval of the RFC and the actual implementation work, a revised design was proposed and written
+up on the tracking issue. During the course of implementing that, it was also discovered that an
+implementation was possible without modifying the number of arguments in a function's MIR, which
+would simplify later stages and unlock use in traits.
+Because the RFC's implementation strategy could not readily support traits, the semantics were not
+originally specified. They have since been implemented following the path which seemed most correct
+to the author and reviewers.
+The basic concept of PGO is to collect data about the typical execution of
+a program (e.g. which branches it is likely to take) and then use this data
+to inform optimizations such as inlining, machine-code layout,
+register allocation, etc.
+There are different ways of collecting data about a program's execution.
+One is to run the program inside a profiler (such as perf) and another
+is to create an instrumented binary, that is, a binary that has data
+collection built into it, and run that.
+The latter usually provides more accurate data.
+So, we are dealing with an instrumentation-based approach, i.e. profiling data
+is generated by a specially instrumented version of the program that's being
+optimized. Instrumentation-based PGO has two components: a compile-time
+component and run-time component, and one needs to understand the overall
+workflow to see how they interact.
+Depending on which step in the above workflow we are in, two different things
+can happen at compile time:
+In the final step of the workflow described above, the program is compiled
+again, with the compiler using the gathered profiling data in order to drive
+optimization decisions. rustc again leaves most of the work to LLVM here,
+basically just telling the LLVM PassManagerBuilder
+where the profiling data can be found:
+LLVM does the rest (e.g. setting branch weights, marking functions with
+cold or inlinehint, etc).
+Instrumentation-based approaches always also have a runtime component, i.e.
+once we have an instrumented program, that program needs to be run in order
+to generate profiling data, and collecting and persisting this profiling
+data needs some infrastructure in place.
+Since the PGO workflow spans multiple compiler invocations most testing happens
+in run-make tests (the relevant tests have pgo in their name).
+There is also a codegen test that checks that some expected
+instrumentation artifacts show up in LLVM IR.
+Clang's documentation contains a good overview on PGO in LLVM here:
+https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization
+The implementation of sanitizers relies almost entirely on LLVM. The rustc is
+an integration point for LLVM compile time instrumentation passes and runtime
+libraries. Highlight of the most important aspects of the implementation:
+This document explains the state of debugging tools support in the Rust compiler (rustc).
+The document gives an overview of debugging tools like GDB, LLDB etc. and infrastructure
+around Rust compiler to debug Rust code. If you want to learn how to debug the Rust compiler
+itself, then you must see Debugging the Compiler page.
+Writing a debugger from scratch for a language requires a lot of work, especially if
+debuggers have to be supported on various platforms. GDB and LLDB, however, can be
+extended to support debugging a language. This is the path that Rust has chosen.
+This document's main goal is to document the said debuggers support in Rust compiler.
+DWARF reader is a program that consumes the DWARF format and creates debugger compatible output.
+This program may live in the compiler itself. DWARF uses a data structure called
+Debugging Information Entry (DIE) which stores the information as "tags" to denote functions,
+variables etc., e.g., DW_TAG_variable, DW_TAG_pointer_type, DW_TAG_subprogram etc.
+You can also invent your own tags and attributes.
+To be able to show debug output we need an expression parser.
+This (GDB) expression parser is written in Bison and is only a subset of Rust expressions.
+This means that this parser can parse only a subset of Rust expressions.
+GDB parser was written from scratch and has no relation to any other parser.
+For example, this parser is not related to Rustc's parser.
+GDB has Rust like value and type output. It can print values and types in a way
+that look like Rust syntax in the output. Or when you print a type as ptype in GDB,
+it also looks like Rust source code. Checkout the documentation in the manual for GDB/Rust.
+Expression parser has a couple of extensions in it to facilitate features that you cannot do
+with Rust. Some limitations are listed in the manual for GDB/Rust. There is some special
+code in the DWARF reader in GDB to support the extensions.
+LLDB currently only works on macOS because of a dependency issue. This issue was easier to
+solve for macOS as compared to Linux. However, Tom has a possible solution which can enable
+us to ship LLDB everywhere.
+There is some special code in the DWARF reader in LLDB to support the extensions.
+A couple of examples of DWARF reader support needed are as follows -
+We have some DWARF extensions that the Rust compiler emits and the debuggers understand that
+are not in the DWARF standard.
+DWARF relies on debuggers to know some information about platform ABI.
+Rust does not do that all the time.
+This section is from the talk about certain aspects of development.
+Shipping GDB requires change to Rustup delivery system. To manage Rustup build size and
+times we need to build GDB separately, on its own and somehow provide the artifacts produced
+to be included in the final build. However, if we can ship GDB with rustup, it will simplify
+the development process by having compiler emit new debug info which can be readily consumed.
+Main issue in achieving this is setting up dependencies. One such dependency is Python. That
+is why we have our own fork of GDB because one of the drivers is patched on Rust's side to
+check the correct version of Python (Python 2.7 in this case. Note: Python3 is not chosen
+for this purpose because Python's stable ABI is limited and is not sufficient for GDB's needs.
+See https://docs.python.org/3/c-api/stable.html).
+This is to keep updates to debugger as fast as possible as we make changes to the debugging symbols.
+In essence, to ship the debugger as soon as new debugging info is added. GDB only releases
+every six months or so. However, the changes that are
+not related to Rust itself should ideally be first merged to upstream eventually.
+We may need to sign up with Apple and get the keys to do this signing. Tom has looked into if
+Mozilla cannot do this because it is at the maximum number of
+keys it is allowed to sign. Tom does not know if Mozilla could get more keys.
+Alternatively, Tom suggests that maybe a Rust legal entity is needed to get the keys via Apple.
+This problem is not technical in nature. If we had such a key we could sign GDB as well and
+ship that.
+Rust traits are not emitted into DWARF at all. The impact of this is calling a method x.method()
+does not work as is. The reason being that method is implemented by a trait, as opposed
+to a type. That information is not present so finding trait methods is missing.
+DWARF has a notion of interface types (possibly added for Java). Tom's idea was to use this
+interface type as traits.
+DWARF only deals with concrete names, not the reference types. So, a given implementation of a
+trait for a type would be one of these interfaces (DW_tag_interface type). Also, the type for
+which it is implemented would describe all the interfaces this type implements. This requires a
+DWARF extension.
+LLVM has Debug Info (DI) builders. This is the primary thing that Rust calls into.
+This is why we need to change LLVM first because that is emitted first and not DWARF directly.
+This is a kind of metadata that you construct and hand-off to LLVM. For the Rustc/LLVM hand-off
+some LLVM DI builder methods are called to construct representation of a type.
+A deeply profound question is that how do you actually debug a procedural macro?
+What is the location you emit for a macro expansion? Consider some of the following cases -
+Focus is to let macros decide what to do. This can be achieved by having some kind of attribute
+that lets the macro tell the compiler where the line marker should be. This affects where you
+set the breakpoints and what happens when you step it.
+Both DWARF and CodeView (PDB) support embedding a cryptographic hash of each source file that
+contributed to the associated binary.
+The cryptographic hash can be used by a debugger to verify that the source file matches the
+executable. If the source file does not match, the debugger can provide a warning to the user.
+The hash can also be used to prove that a given source file has not been modified since it was
+used to compile an executable. Because MD5 and SHA1 both have demonstrated vulnerabilities,
+using SHA256 is recommended for this application.
+A default hashing algorithm is set in the target specification. This allows the target to
+specify the best hash available, since not all targets support all hash algorithms.
+DWARF version 5 supports embedding an MD5 hash to validate the source file version in use.
+DWARF 5 - Section 6.2.4.1 opcode DW_LNCT_MD5
+LLVM IR supports MD5 and SHA1 (and SHA256 in LLVM 11+) source file checksums in the DIFile node.
+The MSVC compiler supports embedding MD5, SHA1, or SHA256 hashes in the PDB using the /ZH
+compiler option.
+Clang always embeds an MD5 checksum, though this does not appear in documentation.
+This is an important idea because debuggers by and large do not try to implement type
+inference. You need to be much more explicit when you type into the debugger than your
+actual source code. So, you cannot just copy and paste an expression from your source
+code to debugger and expect the same answer but this would be nice. This can be helped
+by using compiler.
+It is certainly doable but it is a large project. You certainly need a bridge to the
+debugger because the debugger alone has access to the memory. Both GDB (gcc) and LLDB (clang)
+have this feature. LLDB uses Clang to compile code to JIT and GDB can do the same with GCC.
+Both debuggers expression evaluation implement both a superset and a subset of Rust.
+They implement just the expression language but they also add some extensions like GDB has
+convenience variables. Therefore, if you are taking this route then you not only need
+to do this bridge but may have to add some mode to let the compiler understand some extensions.
+This section covers a numbers of common compiler terms that arise in
+this guide. We try to give the general definition while providing some
+Rust-specific context.
+A control-flow graph is a common term from compilers. If you've ever
+used a flow-chart, then the concept of a control-flow graph will be
+pretty familiar to you. It's a representation of your program that
+exposes the underlying control flow in a very clear way.
+Many expressions that you are used to in Rust compile down to multiple
+basic blocks. For example, consider an if statement:
+When using a control-flow graph, a loop simply appears as a cycle in
+the graph, and the break keyword translates into a path out of that
+cycle.
+Let's describe the concepts of free vs bound in terms of program
+variables, since that's the thing we're most familiar with.
+So there you have it: a variable "appears free" in some
+expression/statement/whatever if it refers to something defined
+outside of that expressions/statement/whatever. Equivalently, we can
+then refer to the "free variables" of an expression – which is just
+the set of variables that "appear free".
+So what does this have to do with regions? Well, we can apply the
+analogous concept to type and regions. For example, in the type &'a u32, 'a appears free. But in the type for<'a> fn(&'a u32), it
+does not.
+rustc has a lot of important data structures. This is an attempt to give some
+guidance on where to learn more about some of the key data structures of the
+compiler.
+This is a reading list of material relevant to Rust. It includes prior
+research that has - at one time or another - influenced the design of
+Rust, as well as publications about Rust.
+What's a project without a sense of humor? And frankly some of these are
+enlightening?
+
+
+
+
