Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
proposal: spec: allow signed shift counts #19113
Proposal: Permit signed integer values as shift count (language change)
Author: Robert Griesemer
Last updated: 2/15/2017
We propose to change the language spec such that the shift count (the rhs operand in a
See Rationale section below.
We change the language spec regarding shift operations as follows: In the section on Operators, the text:
The right operand in a shift expression must have unsigned integer type or be an untyped constant that can be converted to unsigned integer type.
The right operand in a shift expression must have integer type or be an untyped constant that can be converted to an integer type. If the right operand is constant, it must not be negative.
Furthermore, in the section on Integer operators, we change the text:
The shift operators shift the left operand by the shift count specified by the right operand.
The shift operators shift the left operand by the shift count specified by the right operand. A run-time panic occurs if a non-constant shift count is negative.
Since Go's inception, shift counts had to be of unsigned integer type (or a non-negative constant representable as an unsigned integer). The idea behind this rule was that a) the spec didn't have to explain what happened for negative values, and b) the implementation didn't have to deal with negative values possibly occurring at run-time.
In retrospect, this may have been a mistake; a sentiment most recently expressed by @rsc here. It turns out that we could actually change the spec in a backward-compatible way in this regard, and this proposal is suggesting that we do that.
There are other language features where the result (
If we permit any integer type, the existing code will continue to work. Places where we currently use a
An investigation of shifts in the current std library and tests (excluding package-external tests) as of 2/15/2017 (this includes the proposed math/bits package) shows that we have:
If we only look at shifts outside of test files we have:
The overwhelming majority (90%) of shifts outside of testing code is by constant values, and none of those turns out to require a conversion. This proposal won't affect that code.
From the remaining 10% of all shifts, 38% (i.e., 3.8% of all shifts) require a
In this case,
Another one is (cmd/compile/internal/gc/esc.go:1417):
Or this one (/Users/gri/go/src/fmt/scan.go:613):
Many (most?) of the non-constant shifts that don't use an explicit
The drawback of permitting signed integers where negative values are not permitted is that we need to check for them (negative values) at run-time and (most probably) panic, as we do elsewhere (e.g., for
However, none of the existing code will incur that cost because all shift counts are unsigned integers at this point, thus the compiler can omit the check. For new code using non-constant integer shift counts, often the compiler may be able to prove that the operand is non-negative. Furthermore, we can always introduce an explicit
On the plus side, code that used a
This is a backward-compatible language change: Any valid program will continue to be valid, and will continue to run exactly the same, without any performance impact. New programs may be using non-constant integer shift counts as right operands in shift operations. Except for fairly small changes to the spec, the compiler, and go/types, (and possibly go/vet and go/lint if they look at shift operations), no other code needs to be changed.
There's a (remote) chance that some code makes use of negative shift count values:
Open issues (if applicable)
Is there any other case in Go where casting a type generates better/worse code? I think it would be surprising to users that
Moreover, this also means that, eventually, there will be more code using
@rasky There are other situations: For instance in
On the other hand, if we know that
Anybody dealing with performance critical code will already be aware of this (must be aware of the above, for instance, in math/big). For anybody else, this is a net-win because it may actually expose bugs that are now silently swept under the rug via an unchecked
I support this change. I use variable shift a lot and the unsigned shift count requirement is pretty annoying (and also lead to bug sometime: to minimise uint conversion, I sometimes use unsigned loop indices, and get bite when iterating to 0.) As for the performance degradation, if we layout the code properly, the processor should be able to predict the branch correctly every time, so the performance overhead should be minimal.
referenced this issue
Feb 16, 2017
I don't think the problem this solves has been explained well.
The current situation is very clear. Yes, it requires some conversions but in my experience they are not common. For that matter, shifts are not common.
Also, if you permit ints, you allow negative values. Now, you could just panic but many expect a negative shift to reverse the direction of the shift, which is not a good design in practice. People will be confused.
So requiring uint eliminates the consideration completely, which I remember being the main point in Go's decision.
@robpike As you state, requiring uint eliminates the consideration completely, and which is why I also was a strong supporter of the original Go decision. But I am not so convinced anymore that was the best decision looking at actual data.
As it turns out, much of the code (at least in the std lib) that uses non-constants shifts simply uses an unchecked
When you say unchecked conversion, you mean that there is no explicit check that the number is not negative? I don't fully buy this point. If there's some calculation computing a shift amount, your change would allow to catch the subset of bugs which cause the number to be negative (either by really ending up negative, or by becoming negative through overflow); you would still be missing all errors in which the number is just bigger than expected (but not big enough to overflow); so for instance you would still miss most of the off-by-ones, or wrong multiplications by 2/4/8 wrt to the width of the integer being shifted.
I wonder how many hidden bugs of this kind there really are. In fact, shifting by a wrong amount doesn't sound like something that can go unnoticed for a long time, but maybe I just can't picture enough scenarios right now.
I would say that the bug-catching benefit is the one that sounds less convincing to me. The most convincing is making the language simpler for the programmer in the average case (though the runtime cost is hard to swallow -- but I write a lot of performance sensitive code, so I'm sure I'm biased).
@rasky: Answering your respective points:
I haven't thought a lot about this. It seems like there are three options:
We have 1 now; the proposal above is for 3. I think we should evaluate 2 as well. The rationale would be that if the motivation for dropping the conversion is convenience, then make the change only about convenience, not also a semantic change. That would introduce no new overhead or behaviors.
Not advocating for 2, just saying that we should evaluate it along with 3.
Another comparison is integer division and modulo, which are implemented with runtime checks for divisor==0. (Though I'm not sure why, since I see there's also code in runtime to handle SIGFPE.)
We also already have to generate extra code for variable shifts, because the machine instructions ignore the higher order bits of the shift register. (I.e., otherwise
I think the main point would be to remove the argument that can be made for certain APIs returning uint instead of int in order to be more convenient to expected use in shifts. This happened in math/big and almost happened in math/bits.
It's also just a little less fiddly: if everyone is just blindly converting without thinking about it anyway, there's no point to making people do it. It becomes a tiny "debate with a compiler".
@robpike: Shifts are not as uncommon as one might think. Again, using the std lib as reference (incl. _test.go files), I see:
As expected, there's more additions, but only about 56% more. In contrast, there are 5.5 times fewer make calls. Yet we permit an
Of course, for
The same argument (I believe now) can be made for shifts: In all the cases I've been looking at, it's either necessary to add a conversion, or APIs need to be carefully crafted just the right way using
changed the title from
Proposal: Permit integer values as shift count (language change)
proposal: spec: allow signed shift counts
Jun 16, 2017
referenced this issue
Oct 10, 2018
I'm not a big fan of the proposed change as it will create this strange behavior that shifts are faster if you use uints, which seems obscure in my opinion. The potential effect I see on this is that people will probably write using ints to try things out, and move to uints once it's ready anyways (because of performance).
I think option 2 proposed by @rsc is best, as it allows you to use ints, but does not present the same problem.
@ianlancetaylor in the case of uints you know the value is positive and therefore you don't need a runtime check for it. When the shift is a variable int, you need to check at runtime that the value is non-negative. This extra check would make it in general more desirable (from a performance point of view) to use the uint version.
The compiler has a "prove" pass that track an integer's bounds. For example:
I'm sure that the compiler could reuse this knowledge to avoid checks on most signed integer shifts. It already does that for index expressions, where a negative value is always a panic.
We already need to check at run time whether the shift count is larger than the size of the value being shifted. Checking for a negative value can be done in the same instruction using an unsigned comparison. If we need to distinguish negative and too-large, we can do that after the single comparison. The normal case of a valid value will run at the same speed as before.
Except that we currently use masking tricks instead of branches to handle large shift counts (at least on AMD64), so while the comparison will be free, there will still be a new conditional branch in the happy path.
That said, I think the performance/readability concern here already exists. For example, many existing shifts can be made faster by masking the shift amount, to aid the compiler in proving that the shift amount is smaller than the word size. And exactly what the compiler can and cannot prove w.r.t. the shift amount varies significantly already. So I'm not sure allowing ints here makes things worse in the normal case. And as always, premature micro-optimization will hurt readability.