Make fields of `Span` private #43968

petrochenkov · Aug 18, 2017

I actually tried to intern spans and benchmark the result^*, and this was a prerequisite.
This kind of encapsulation will be a prerequisite for any other attempt to compress span's representation, so I decided to submit this change alone.

The issue #43088 seems relevant, but it looks like SpanId won't be able to reuse this interface, unless the tables are global (like interner that I tried) and are not a part of HIR.
r? @michaelwoerister anyway

^* Interning means 2-3 times more space is required for a single span, but duplicates are free. In practice it turned out that duplicates are not that common, so more memory was wasted by interning rather than saved.

petrochenkov · Aug 18, 2017

petrochenkov reviewed Aug 18, 2017

View changes

petrochenkov · Aug 18, 2017

src/libsyntax_pos/lib.rs

+    }
+    #[inline]
+    #[must_use]
+    pub fn set_lo(self, lo: BytePos) -> Span {


I'm not sure set_x are the best possible names since the functions are not mutating, but at least I guarded them from mistakes with #[must_use].

Should the fields be named with_x? In my mind that conjures the fact that the method returns a copy of the same type with the field having the new value.

Yeah, seems better, I'll rename.

kennytm · Aug 18, 2017

kennytm reviewed Aug 18, 2017

View changes

kennytm · Aug 18, 2017

src/librustc/hir/lowering.rs

@@ -425,7 +425,7 @@ impl<'a> LoweringContext<'a> {
                allow_internal_unsafe: false,
            },
        });
-        span.ctxt = SyntaxContext::empty().apply_mark(mark);
+        span = span.set_ctxt(SyntaxContext::empty().apply_mark(mark));
        span


Huh? Just return span.set_ctxt(SyntaxContext::empty().apply_mark(mark)) directly.

estebank · Aug 18, 2017

estebank reviewed Aug 18, 2017

View changes

estebank · Aug 18, 2017

src/libsyntax_pos/lib.rs

+    #[inline]
+    #[must_use]
+    pub fn set_lo(self, lo: BytePos) -> Span {
+        Span::new(lo, self.hi(), self.ctxt())


Wouldn't it make more sense to do the following in all the set_x methods?

Span {lo, ..self}

It was necessary with interning (the same is true for #43968 (comment)). It will be necessary with any non-trivial Span::new/Span::lo/etc, so I left it.

estebank · Aug 18, 2017

src/libsyntax_pos/lib.rs

+    }
+    #[inline]
+    #[must_use]
+    pub fn set_lo(self, lo: BytePos) -> Span {


Should the fields be named with_x? In my mind that conjures the fact that the method returns a copy of the same type with the field having the new value.

estebank · Aug 18, 2017

src/libsyntax_pos/lib.rs

-        let lo = cmp::max(self.hi.0 - 1, self.lo.0);
-        Span { lo: BytePos(lo), ..self }
+        let lo = cmp::max(self.hi().0 - 1, self.lo().0);
+        self.set_lo(BytePos(lo))


Is it really necessary to use the public getters in the impl?

I guess it would be if spans were really interned. For now I'd also opt for just accessing the fields directly.

michaelwoerister · Aug 18, 2017

michaelwoerister reviewed Aug 18, 2017

View changes

Looks good to me. Thanks, @petrochenkov! I'm also for renaming the set_x methods to with_x. Other than that (and maybe not using the getters in impl Span) this is good to go.

You might be interested in this: A couple of years ago I did some experiments with span interning plus storing the span information directly in the interning key, if it fit in there (like a tagged pointer). The results were actually quite promising, if I remember correctly: https://internals.rust-lang.org/t/rfc-compiler-refactoring-spans/1357/23. Maybe you want to revisit interning with this additional optimization?

michaelwoerister · Aug 18, 2017

src/libsyntax_pos/lib.rs

-        let lo = cmp::max(self.hi.0 - 1, self.lo.0);
-        Span { lo: BytePos(lo), ..self }
+        let lo = cmp::max(self.hi().0 - 1, self.lo().0);
+        self.set_lo(BytePos(lo))


I guess it would be if spans were really interned. For now I'd also opt for just accessing the fields directly.

petrochenkov · Aug 18, 2017

@michaelwoerister

The results were actually quite promising, if I remember correctly: https://internals.rust-lang.org/t/rfc-compiler-refactoring-spans/1357/23. Maybe you want to revisit interning with this additional optimization?

I remembered there was a thread about this somewhere! Thanks for the link.
Yes, I was going to experiment further with keeping "short" spans inline and other spans "somewhere else".

petrochenkov · Aug 18, 2017

Updated.

estebank · Aug 19, 2017

estebank reviewed Aug 19, 2017

View changes

estebank · Aug 19, 2017

src/libsyntax/codemap.rs

+            Some(Span::new(
+                cmp::min(sp_lhs.lo(), sp_rhs.lo()),
+                cmp::max(sp_lhs.hi(), sp_rhs.hi()),
+                sp_lhs.ctxt()


I believe this can be replaced with Some(sp_lhs.to(sp_rhs)).

bors · Aug 19, 2017

☔️ The latest upstream changes (presumably #43933) made this pull request unmergeable. Please resolve the merge conflicts.

michaelwoerister · Aug 19, 2017

Thanks! r=me after rebasing.

petrochenkov · Aug 19, 2017

@bors r=michaelwoerister

bors · Aug 19, 2017

📌 Commit cdae234 has been approved by michaelwoerister

bors · Aug 19, 2017

⌛️ Testing commit cdae234 with merge 17e924e...

bors · Aug 19, 2017

💔 Test failed - status-travis

petrochenkov · Aug 19, 2017

Sigh, need to send a PR to rustfmt first.

kennytm · Aug 20, 2017

We've hit a deadlock situation where the rustfmt PR cannot pass without the get/set methods added in this PR, and this PR cannot pass without rustfmt stop reading the now-private fields. I don't think turning rustfmt's status red is a good idea.

I think this PR should be split into two stages,

This PR keep the fields public, so rustfmt doesn't need to be updated at the moment.
rustfmt (as well as clippy, rls, etc.) all accept PRs stop using the fields.
Submit another PR to turn the fields private (can be done simultaneously with step 2).

petrochenkov · Aug 20, 2017

@kennytm
Situations like this were previously resolved by creating a new branch in rustfmt/rls, merging the breaking patch into that branch and not master, then setting rls submodule in rust-lang/rust to that branch.

petrochenkov · Aug 20, 2017

I gathered some span statistics from rustc/libstd and found a significant number of "reverse" spans with lo > hi!
That seems bad, some code may work correctly with such spans, but I've certainly seen ICEs caused by code that doesn't work with them.

I'm going to add "normalization" to lo <= hi into Span::new.
It also boosts the percent of inline spans a bit when compression is used.

petrochenkov · Aug 20, 2017

@bors r=michaelwoerister

bors · Aug 20, 2017

📌 Commit 3da163f has been approved by michaelwoerister

bors · Aug 20, 2017

⌛️ Testing commit 3da163f with merge 4ad2c42...

bors · Aug 21, 2017

💔 Test failed - status-travis

petrochenkov · Aug 21, 2017

@bors r=michaelwoerister

bors · Aug 21, 2017

📌 Commit c4125e2 has been approved by michaelwoerister

bors · Aug 21, 2017

⌛️ Testing commit c4125e2 with merge c1ad755...

bors · Aug 21, 2017

💔 Test failed - status-travis

kennytm · Aug 21, 2017

x86_64-gnu-aux, same error. It's legit then.

nrc · Aug 22, 2017

RLS tests timed out or some span is not found. Maybe legit

Possibly a bad span produced in save-analysis?

petrochenkov · Aug 22, 2017

The errors are caused by RLS update, not by span patches, i.e. 9061581 alone (without rust-lang/rls@f4e6f16) is enough to reproduce them.
In theory, #44028 should hit them as well.
cc @nrc

estebank · Aug 22, 2017

Could you provide a dump of the reversed spans? I've seen them every now and then but have put off investigating the root cause(s). There are a few possible very unseemly results that could happen, for example when doing end.to(start), so I'd hate to just mask the incorrect spans by doing the obvious fix but rather fix the source (probably during parsing).

petrochenkov · Aug 23, 2017

Waiting on #44028

petrochenkov · Aug 23, 2017

@estebank

Could you provide a dump of the reversed spans?

I have only numbers unfortunately. Looks like the only way to obtain the source of inverted spans from Span::new is backtrace.
Once this PR lands, Span::new will be the single point of span creation, so you will be able to detect inverted spans by inserting an assert(lo <= hi) into it and building everything with RUST_BACKTRACE=1.
EDIT: Maybe that assert could be left there forever once all inverted spans are fixed.

estebank · Aug 24, 2017

Sounds reasonable. Will do. Are asserts only enabled on nightly? Would rather not cause ICEs for a rather benign bug just because it isn't caught in the test, but would like to see he effects of it on cargo bomb.

bors · Aug 26, 2017

☔️ The latest upstream changes (presumably #44028) made this pull request unmergeable. Please resolve the merge conflicts.

petrochenkov · Aug 26, 2017

@estebank

Are asserts only enabled on nightly?

Not sure about debug_assert, but assert is always enabled.
FWIW, nightly can be detected through an environmental variable CFG_RELEASE_CHANNEL, but I'm not sure it's a good idea.

petrochenkov · Aug 29, 2017

I'll try to land changes to rustc and rustfmt in two steps (rustfmt is moving forward too fast), so I made fields of Span public again for compatibility.
@bors r=michaelwoerister

bors · Aug 29, 2017

📌 Commit a0c3264 has been approved by michaelwoerister

bors · Aug 30, 2017

⌛️ Testing commit a0c3264 with merge ca9cf35...

bors · Aug 30, 2017

☀️ Test successful - status-appveyor, status-travis
Approved by: michaelwoerister
Pushing ca9cf35 to master...

rust-highfive assigned michaelwoerister Aug 18, 2017

petrochenkov force-pushed the petrochenkov:span2 branch from 2c1d664 to 9fed6da Aug 18, 2017

mcarton referenced this pull request Aug 19, 2017
Closed
Breakage incoming #1972

petrochenkov force-pushed the petrochenkov:span2 branch from 9fed6da to cdae234 Aug 19, 2017

petrochenkov referenced this pull request Aug 19, 2017
Closed
Use getters to access `Span` fields #1899

nrc added a commit to rust-lang/rls that referenced this pull request Aug 20, 2017

Depend on the `span` branch of rustfmt
to land rust-lang/rust#43968

Loading status checks…

f4e6f16

petrochenkov force-pushed the petrochenkov:span2 branch from cdae234 to 3da163f Aug 20, 2017

nrc referenced this pull request Aug 22, 2017
Merged
Documents managing tools in the Rust repo #27

arielb1 added the S-waiting-on-author label Aug 22, 2017

nrc added a commit to rust-lang/rls that referenced this pull request Aug 27, 2017

Depend on the `span` branch of rustfmt
to land rust-lang/rust#43968

Loading status checks…

903cb9f

petrochenkov added some commits Jul 31, 2017

Make fields of `Span` private

3da868d

Normalize order of `lo` and `hi` in `Span::new`

71dfe64

Make fields of `Span` public again
This helps to avoid landing changes to rustc and rustfmt in one step

Loading status checks…

a0c3264

petrochenkov force-pushed the petrochenkov:span2 branch from c4125e2 to a0c3264 Aug 29, 2017

bors merged commit a0c3264 into rust-lang:master Aug 30, 2017
2 checks passed

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

homu Test successful
Details

This was referenced Aug 30, 2017

Merged

Use hir::ItemLocalId instead of ast::NodeId in rustc::middle::region::CodeExtent. #44171

Closed

Add expansion info to crate metadata #43847

petrochenkov deleted the petrochenkov:span2 branch Sep 10, 2017

rust-lang/rust

Join GitHub today

Make fields of `Span` private #43968

Conversation

petrochenkov commented Aug 18, 2017 • edited

rust-highfive assigned michaelwoerister Aug 18, 2017

petrochenkov reviewed Aug 18, 2017 View changes

This comment has been minimized.

petrochenkov Aug 18, 2017 • edited

This comment has been minimized.

This comment has been minimized.

kennytm reviewed Aug 18, 2017 View changes

This comment has been minimized.

estebank reviewed Aug 18, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

michaelwoerister reviewed Aug 18, 2017 View changes

michaelwoerister left a comment

This comment has been minimized.

This comment has been minimized.

petrochenkov commented Aug 18, 2017

petrochenkov force-pushed the petrochenkov:span2 branch from 2c1d664 to 9fed6da Aug 18, 2017

This comment has been minimized.

petrochenkov commented Aug 18, 2017

estebank reviewed Aug 19, 2017 View changes

This comment has been minimized.

This comment has been minimized.

bors commented Aug 19, 2017

This comment has been minimized.

michaelwoerister commented Aug 19, 2017

mcarton referenced this pull request Aug 19, 2017

petrochenkov force-pushed the petrochenkov:span2 branch from 9fed6da to cdae234 Aug 19, 2017

This comment has been minimized.

petrochenkov commented Aug 19, 2017

This comment has been minimized.

bors commented Aug 19, 2017

This comment has been minimized.

bors commented Aug 19, 2017

bors added a commit that referenced this pull request Aug 19, 2017

This comment has been minimized.

bors commented Aug 19, 2017

This comment has been minimized.

petrochenkov commented Aug 19, 2017

petrochenkov referenced this pull request Aug 19, 2017

This comment has been minimized.

kennytm commented Aug 20, 2017

This comment has been minimized.

petrochenkov commented Aug 20, 2017

This comment has been minimized.

petrochenkov commented Aug 20, 2017

nrc added a commit to rust-lang/rls that referenced this pull request Aug 20, 2017

petrochenkov force-pushed the petrochenkov:span2 branch from cdae234 to 3da163f Aug 20, 2017

This comment has been minimized.

petrochenkov commented Aug 20, 2017

This comment has been minimized.

bors commented Aug 20, 2017

This comment has been minimized.

bors commented Aug 20, 2017

bors added a commit that referenced this pull request Aug 20, 2017

This comment has been minimized.

bors commented Aug 21, 2017

This comment has been minimized.

petrochenkov commented Aug 21, 2017

This comment has been minimized.

bors commented Aug 21, 2017

This comment has been minimized.

bors commented Aug 21, 2017

bors added a commit that referenced this pull request Aug 21, 2017

This comment has been minimized.

bors commented Aug 21, 2017

This comment has been minimized.

kennytm commented Aug 21, 2017

This comment has been minimized.

nrc commented Aug 22, 2017

This comment has been minimized.

petrochenkov commented Aug 22, 2017 • edited

nrc referenced this pull request Aug 22, 2017

petrochenkov commented Aug 18, 2017 •

edited

petrochenkov reviewed Aug 18, 2017

View changes

petrochenkov Aug 18, 2017 •

edited

kennytm reviewed Aug 18, 2017

View changes

estebank reviewed Aug 18, 2017

View changes

michaelwoerister reviewed Aug 18, 2017

View changes

petrochenkov force-pushed the petrochenkov:span2 branch from `2c1d664` to `9fed6da` Aug 18, 2017

estebank reviewed Aug 19, 2017

View changes

petrochenkov force-pushed the petrochenkov:span2 branch from `9fed6da` to `cdae234` Aug 19, 2017

petrochenkov force-pushed the petrochenkov:span2 branch from `cdae234` to `3da163f` Aug 20, 2017

petrochenkov commented Aug 22, 2017 •

edited

petrochenkov commented Aug 23, 2017 •

edited

petrochenkov commented Aug 26, 2017 •

edited

petrochenkov force-pushed the petrochenkov:span2 branch from `c4125e2` to `a0c3264` Aug 29, 2017

bors merged commit `a0c3264` into rust-lang:master Aug 30, 2017
2 checks passed