New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: spec: binary integer literals #19308

Open
EddieRingle opened this Issue Feb 27, 2017 · 87 comments

Comments

Projects
None yet
@EddieRingle
Copy link

EddieRingle commented Feb 27, 2017

Currently, Go supports octal and hexadecimal integer literals in addition to the standard decimal literals you'd expect. In an effort to round out this group, I propose adding binary integer literals as well. They would come in the form as a new prefix to integer literals: 0b or 0B.

Initial Disclaimer

This was my first dive into the Go source code and primarily a learning experience for me to get my feet wet submitting changes and what-not. That being said, I appreciate any and all criticism, comments, and suggestions.

The "Why": Prior Art

Binary literals exist or have appeared in mainstream languages as well, including:

All of the above cases have settled on a convention of using 0b or 0B to prefix binary literals, suggesting that this would be a fairly comfortable and sensible choice for Go as well, avoiding needless invention and providing similarity for programmers coming from these other aforementioned languages.

The "Why": Continued

I managed to find some earlier discussions relating to this, albeit this was after I had already implemented the feature and it has more to do with changing the octal syntax than specifically relating to binary literals. But #12711 (comment) from @griesemer does mention that "both the 'o' for octal and 'b' for binary notation were also discussed in the design of Go. It's simply not enough bang for the buck." However, I don't see this as a good argument against adding something so simple to the language. Especially considering the fact that nowadays more and more languages have adopted the syntax for binary literals, it seems the earlier Go design decisions might need to be looked at under new light.

Example usage

const (
    SOME_MASK   = 0b00001111
    SOME_FLAG_A = 0b00000001
    SOME_FLAG_B = 0b00000010
    SOME_FLAG_C = 0b00000100
    SOME_FLAG_D = 0b00001000
)

Implementation

As I stated this was more of a learning experience for me, I already have changes that implement this feature ready:

CL-37502 spec: specify syntax for binary integer literals
CL-37503 cmd/compile/internal/syntax: scan binary literals
CL-37504 go/scanner: scan binary integer literals
CL-37505 strconv: support parsing binary integer literals
CL-37506 test: extend int_lit with binary literal usages

@bradfitz bradfitz added this to the Proposal milestone Feb 27, 2017

@bradfitz bradfitz added the Proposal label Feb 27, 2017

@griesemer

This comment has been minimized.

Copy link
Contributor

griesemer commented Feb 27, 2017

This has come up before. There is a significant amount of work involved in rolling this out, making the compiler and spec changes is trivial. But there's a lot of libraries that should also be made consistent (strconv, math/big, etc.).

If we make a change in this direction, it should be more thorough and support arbitrary bases. I am against this as is.

@EddieRingle

This comment has been minimized.

Copy link
Author

EddieRingle commented Feb 27, 2017

@griesemer Yes, the changes I'm about to submit also modifies strconv (to my understanding, it's actually required to support this change).

@EddieRingle

This comment has been minimized.

Copy link
Author

EddieRingle commented Feb 27, 2017

@griesemer I disagree however that any change should support arbitrary bases or otherwise no change be made at all. From prior reading, it sounds like that would be a good goal for Go2; this is simply polishing Go1 with syntax developers of other languages might expect when using Go. (i.e. Base-2 is a common enough case, probably more common than octal; base-14 or what-have-you is less-so.)

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 27, 2017

CL https://golang.org/cl/37503 mentions this issue.

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 27, 2017

CL https://golang.org/cl/37504 mentions this issue.

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 27, 2017

CL https://golang.org/cl/37502 mentions this issue.

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 27, 2017

CL https://golang.org/cl/37505 mentions this issue.

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 27, 2017

CL https://golang.org/cl/37506 mentions this issue.

@ALTree

This comment has been minimized.

Copy link
Member

ALTree commented Feb 27, 2017

However, I don't see this as a good argument against adding something so simple to the language.

It's not a particularly strong argument in favour of adding them, either.

IMHO you'll need to expand the "why" section explaining exactly what advantages supporting binary literals will bring to people writing go code.

I don't find them particularly useful; hex is a much more readable and compact format for literals that have a "bit-level meaning".

You gave an 'usage example', but it's not very compelling. I would write those constants using 0xfs and shifts for the others.

@griesemer

This comment has been minimized.

Copy link
Contributor

griesemer commented Feb 27, 2017

@EddieRingle This proposal has not been discussed widely nor has it been accepted. Please do not spam us with code reviews. The Go Team has enough to do with work that's actually important.

It is clear to everybody that adding a simple feature to the language is trivial. It's also clear that plenty of people would like this feature (I myself would have liked it sometime). But that said, just because one can, is not an argument that one should. Any however small and simple addition to a language has long-term costs. If we accept this, it will become even more difficult in the future to have a more general mechanism, and we need to remain backward compatible.

Let's wait and see what other people say before jumping the gun. Thanks.

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 27, 2017

Reminder of our no-me-too policy: https://golang.org/wiki/NoMeToo

Opinions without constructive content can be expressed using Github's emoji reactions.

@EddieRingle

This comment has been minimized.

Copy link
Author

EddieRingle commented Feb 27, 2017

@ALTree

IMHO you'll need to expand the "why" section explaining exactly what advantages supporting binary literals will bring to people writing go code.

I don't find them particularly useful; hex is a much more readable and compact format for literals that have a "bit-level meaning", IMO.

I'd argue the opposite, actually. Hex is more compact in many cases, yes, but binary literals would be an exact "bit-level" representation and therefore as readable as you could get.

@griesemer

This proposal has not been discussed widely nor has it been accepted. Please do not spam us with code reviews. The Go Team has enough to do with work that's actually important.

Apologies. It was originally a single change but since it seems like the Go policy is to break up commits based on the area of codebase affected, that's how I ended up splitting them up. I wasn't aware the bot would make individual comments here for every change. I wouldn't be so cold as to call it spamming you, however, nor imply that any effort I put in using my free time is unimportant.

Any however small and simple addition to a language has long-term costs. If we accept this, it will become even more difficult in the future to have a more general mechanism, and we need to remain backward compatible.

Like was mentioned before, the general-purpose route (which I would prefer, as well) would also encourage the deprecation/removal of the existing (confusing) octal syntax, no? The feeling I got was that the general-purpose syntax (2r0010 or 2x0010 for base-2, for example) was invention meant for Go2, where breaking changes would be welcome anyway.

Putting a potential Go2 aside, to address the statement that "if we accept this, it will become even more difficult in the future to have a more general mechanism": I just don't see how this is true. Adding the binary literal prefix would be orthogonal to an alternative, general-purpose syntax, especially the one you described in #12711 (in fact, that syntax conflicts directly with hexadecimal literals, but would not with this proposed binary literal syntax). They would exist side-by-side just as the general-purpose syntax would with the existing octal, hex, and decimal literals.

@cespare

This comment has been minimized.

Copy link
Contributor

cespare commented Feb 27, 2017

Apologies. It was originally a single change but since it seems like the Go policy is to break up commits based on the area of codebase affected, that's how I ended up splitting them up. I wasn't aware the bot would make individual comments here for every change. I wouldn't be so cold as to call it spamming you, however, nor imply that any effort I put in using my free time is unimportant.

It's not just that the bot sends mail about CLs, it's that each mailed CL is a request for a Go reviewer to spend time reviewing it.

@wedow

This comment has been minimized.

Copy link

wedow commented Feb 27, 2017

The 0b syntax is nice because it's familiar but if the true goal is simply to add binary literals to the language, I would much prefer the generic solution over the familiar.

Is there any technical reason the generic option can't be implemented prior to 2.0? I've had a number of cases lately where binary literals would have been preferred over hex and it would be nice to have that option in 1.9 or 1.10 instead of waiting (possibly many years) until 2.0.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Feb 27, 2017

@wedow I think it would help to see specific real cases where binary literals are useful. Please share the cases where binary literals would be helpful. Thanks.

@dr2chase

This comment has been minimized.

Copy link
Contributor

dr2chase commented Feb 27, 2017

I don't see "should support arbitrary bases" as a worthwhile objection. It adds complexity/cost for little or not additional benefit. Over all the years I've been hacking, the would-be-useful bases I've heard of people wanting to use are 2, 8, 10, 12, and 16, and possibly 64 (we have base64 encoding, after all).

@rsc rsc changed the title proposal: Binary integer literals proposal: spec: binary integer literals Feb 27, 2017

@AndreasBackx

This comment has been minimized.

Copy link

AndreasBackx commented Feb 27, 2017

Let's look at why the aforementioned languages have moved forward and added support for binary literals. Let's start with C++14 as it is the first one on the list. What were the points James Dennett, then Google employee, brought forward?

Use of an 0b/0B prefix for binary literals is an existing GCC extension (also supported by Clang), and is the same syntax as Java 7, Python, and D.

Why would this particular point benefit Go?

Familiarity with the language.

Many developers move from programming language to programming language. Correct me if I'm wrong, but we all try what we know from one language in another one. You could in a way say that Go's familiarity with C and C++ attracted those kinds of developers who wanted a particular feature like GC, but didn't like other languages. This is one of the reason's the company I currently intern for has chosen to move to Go.

Besides experienced developers, let's also have a look what the benefit is of familiarity for beginner developers. Let's also take the use case of flags that @EddieRingle mentioned. Explaining to a complete beginner how flags work is a lot harder when you have to explain it in octal or hexadecimal as it requires the person to learn those as well.

Lastly I would like to add here that the purpose of every language (at least that's what I hope) is to write clean code. And I think that is something we all agree on regardless. When looking over someone else's code, it is immediately clear without any explanation that when having a list of binary literal constants, that those are flags. That same thing is a lot less straightforward when using hexadecimal or octal. Below is a comparison.

// Hexadecimal
const (
    MASK          = 0x1E
    DEFAULT_COLOR = 0x00
    BOLD          = 0x01
    UNDERLINE     = 0x02
    FLASHING_TEXT = 0x04
    NO_CHANGE     = 0x08
)

// Octal
const (
    MASK          = 036
    DEFAULT_COLOR = 00
    BOLD          = 01
    UNDERLINE     = 02
    FLASHING_TEXT = 04
    NO_CHANGE     = 010
)

// Binary
const (
    MASK          = 0b11110
    DEFAULT_COLOR = 0b00000
    BOLD          = 0b00001
    UNDERLINE     = 0b00010
    FLASHING_TEXT = 0b00100
    NO_CHANGE     = 0b01000
)

I do think that no thought needs to be given to the fact that the last constants are used for flags. These are also very few flags, so keep in mind that this definitely adds up when having more flags. The first constant 0x1E can definitely turn some heads when it's declared without context. Using binary literals alone may indicate that a variable might be used as a flag.

The referred C++ PDF further refers to the aforementioned languages for support. So let's look at those next. I have found the (original?) proposal by Derek Foster in 2009 for binary literals in JDK. Source

The first thing it questions, which I completely agree with, is why there is an octal representation in the JDK, but there is no binary representation in the JDK. In the past few years I have never thought to myself: "Oh, octals would make my code cleaner!" This however refers to the previous point I made: familiarity. There is one thing it however adds to my previously made point:

When the data being dealt with is fundamentally bit-oriented, however, using hexadecimal to represent ranges of bits requires an extra degree of translation for the programmer, and this can often become a source of errors. [...] then a programmer coding to that specification must translate each such value from its binary representation into hexadecimal. [...] In most cases, programmers do these translations in their heads, and HOPEFULLY get them right. however, errors can easily creep in, and re-verifying the results is not straightforward enough to be done frequently.

Hexadecimal and octal notation used primarily in hardware instead of binary can result in human errors. In the comparison I gave previously, I checked what I did with my head by entering what I thought was octal into Google which confirmed my answers. I am automatically sure of my case when I write it in binary, but am not when I write it in hexadecimal or octal. And no matter how many times you do this a day, it makes it harder to write code because you have to think of the binary form in your head and while doing so it is possible to make errors.

To dig deeper into the question to why there is an octal notation, but no binary notation I have another question to ask which was also asked by Derek Foster, the author of the binary literal proposal for the JDK: "Why did Go chose to use the 0 prefix for octal notations?" @griesemer commented that we should not jump the gun when implementing new features:

Let's wait and see what other people say before jumping the gun. Thanks.

But hasn't Go jumped the gun when implementing the octal notation? If its argument was "because other languages do it", then why can't that argument not be used for binary literals? If it wasn't, then what was the reason that the 0 prefix for octal notations made it into the language when it confuses people?

Someone might incorrectly think that "0b1" represented the same value as hexadecimal number "0xB1". However, note that this problem has existed for octal/decimal for many years (confusion between "050" and "50") and does not seem to be a major issue.

- Derek Foster

There don't seem to be more points in favour of binary literals because it is something we all refer to in our heads. This is why I feel proposals for other languages have been brief like this one. It however is not a reason to shut it down so quickly.

@randall77

This comment has been minimized.

Copy link
Contributor

randall77 commented Feb 27, 2017

Here's another option, which seems clearer to me than any of the integer constants.

// Shifts
const (
    MASK          = 0x1e
    DEFAULT_COLOR = 0
    BOLD          = 1<<0
    UNDERLINE     = 1<<1
    FLASHING_TEXT = 1<<2
    NO_CHANGE     = 1<<3
)

(And shouldn't mask be 0xf, not 0x1e?)

I'm mildly against adding binary constants, at least in Go 1. I'd be for adding them in Go 2, though. The reason for the difference is that if for whatever reason someone is stuck at Go 1.8, when Go 1.9 comes out with binary constants, if one of the (transitive) imports of that person's code uses binary constants, then they can no longer build their project using Go 1.8. They would have to vendor or upgrade. There's a definite cost to adding a forward-incompatible feature which should weigh against its utility.

I do agree that I don't see any need for bases not in {2,8,10,16}. The case for octal seems particularly shaky, I'd be for removing octal in Go 2.

@AndreasBackx

This comment has been minimized.

Copy link

AndreasBackx commented Feb 27, 2017

@randall77 I disagree that shifting looks cleaner. In my head I'm still representing them as binary numbers and probably always will. It would make it easier to remove that calculation I do in my head.

(And shouldn't mask be 0xf, not 0x1e?)

The name MASK was merely taken from the JDK proposal and is not really in line with the other constants. But it does show that 0x1E and hexadecimals already cause confusion.

I can understand the point you're trying to make wanting to move it to Go 2. But I disagree that we should have to support projects that downgrade their Go version from 1.9 to 1.8. It would make languages changes a nightmare to deal with. I do however not know how Go looks at this, it would be most wise to follow what compatibility Go has in mind.

I wholeheartedly support your position on removing octal in Go 2.

@griesemer

This comment has been minimized.

Copy link
Contributor

griesemer commented Feb 27, 2017

I just reread my previous comment (specifically, "The Go Team has enough to do with work that's actually important."). I want to apologize for this statement which was a rather offensive formulation of what I actually meant to say. So let me try again, elaborating a bit more and hopefully finding the right tone this time:

We do appreciate proposals that are well substantiated and, if necessary, come with prototype implementations. That said, the Go proposal process is light-weight on purpose, and no extra work is required by the proposer unless requested or necessary to understand the proposal. Sending change lists that are not requested and/or don't fix an issue is counter-productive since somebody will have to take the time and look at them (if only to postpone or close them). A better approach, if one does want to prototype/write the code ahead of time, is to link to the changes elsewhere (for instance, a private GitHub commit). This will leave the Go Team and external contributors the choice: They can decide to look at that code if they want to, or otherwise focus on higher-priority items. Thanks.

@EddieRingle

This comment has been minimized.

Copy link
Author

EddieRingle commented Feb 27, 2017

@griesemer Gotcha, I understand and that makes sense. I assumed the Go team treated their Gerrit like AOSP does theirs, and figured my changes could exist there while this was discussed. Linking to a branch here on GitHub is less work anyway, so it's a win-win, I guess. :)

I actually did the work first since my main goal was to hack on the compiler. It was after the fact that I decided to submit it as a proposal.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Feb 27, 2017

@AndreasBackx The leading 0 for octal in Go was discussed in issue #151. See also #12711.

@ALTree

This comment has been minimized.

Copy link
Member

ALTree commented Feb 27, 2017

When defining 1-set-bit constants, shifting is more readable than 0b00001000..00 for the simple reason that in the shift version you don't need to count a bunch of zeros on the screen just to understand which bit is set; you just read the shift value.

0b100000000000000000000000 vs 1 << 23

@driusan

This comment has been minimized.

Copy link

driusan commented Feb 27, 2017

As far as real world usage goes, a common variable length integer encoding method is to use the high bit for "read more". I've had to use it to extract git packfiles. Here's the code to extract the lower bits in various bases:

b & 127
b & 0x1f
b & 0177
b & 0b01111111

I personally think that the binary version more clearly shows the intent.

@bamiaux

This comment has been minimized.

Copy link

bamiaux commented Feb 28, 2017

You still have the shift option mentioned previously
If you think it's not readable, use an helper function

b & ^(^0 << 7)
b & mask(7)
@RalphCorderoy

This comment has been minimized.

Copy link

RalphCorderoy commented Feb 28, 2017

@AndreasBackx, 1<<12 is more clear than 0b0001000000000000 because I don't have to count all those zeroes. It's apparent it's a mask because the 12 falls between 11 and 13, or uses iota. When having to match an arbitrary pattern, e.g. masking bits from an instruction word, then hex is better because a programmer used to dealing with bits can read 0xae and "see" 10101110 by virtue of knowing 0xa, ten, is 1010, mnemonic ten-ten, just as one learns times tables, 65 is ASCII A, etc. Hex is a more dense representation that's easier to parse for the human reader.

@randall77, aren't 0644, 02775, etc., a bit tedious without octal? That's why it's still kicking about.

@randall77

This comment has been minimized.

Copy link
Contributor

randall77 commented Feb 28, 2017

@RalphCorderoy : yes, it seems to me that octal survives for the sole reason of constructing an os.FileMode.
0664 = 6<<6 + 6<<3 + 4, which isn't too tedious. It would be better if os provided symbolic constants to make this easier, or at least clearer.

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 4, 2019

Change https://golang.org/cl/161098 mentions this issue: spec: document new Go2 number literals

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 5, 2019

Change https://golang.org/cl/161199 mentions this issue: text/scanner: accept new Go2 number literals

gopherbot pushed a commit that referenced this issue Feb 11, 2019

cmd/compile: accept new Go2 number literals
This CL introduces compiler support for the new binary and octal integer
literals, hexadecimal floats, and digit separators for all number literals.

The new Go 2 number literal scanner accepts the following liberal format:

number   = [ prefix ] digits [ "." digits ] [ exponent ] [ "i" ] .
prefix   = "0" [ "b" |"B" | "o" | "O" | "x" | "X" ] .
digits   = { digit | "_" } .
exponent = ( "e" | "E" | "p" | "P" ) [ "+" | "-" ] digits .

If the number starts with "0x" or "0X", digit is any hexadecimal digit;
otherwise, digit is any decimal digit. If the accepted number is not valid,
errors are reported accordingly.

See the new test cases in scanner_test.go for a selection of valid and
invalid numbers and the respective error messages.

R=Go1.13

Updates #12711.
Updates #19308.
Updates #28493.
Updates #29008.

Change-Id: Ic8febc7bd4dc5186b16a8c8897691e81125cf0ca
Reviewed-on: https://go-review.googlesource.com/c/157677
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>

gopherbot pushed a commit that referenced this issue Feb 11, 2019

go/scanner: accept new Go2 number literals
This CL introduces go/scanner support for the new binary and octal integer
literals, hexadecimal floats, and digit separators for all number literals.
The new code is closely mirroring the respective code for number literals in
cmd/compile/internal/syntax/scanner.go.

R=Go1.13

Updates #12711.
Updates #19308.
Updates #28493.
Updates #29008.

Change-Id: I5315c6aaa7cfc41a618296be20e3acd5114d6b3c
Reviewed-on: https://go-review.googlesource.com/c/159997
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>

gopherbot pushed a commit that referenced this issue Feb 11, 2019

cmd/gofmt: test that Go 2 number literals can be formatted
R=Go1.13

Updates #12711.
Updates #19308.
Updates #28493.
Updates #29008.

Change-Id: Icd25aa7f6e18ed671ea6cf2b1b292899daf4b1a5
Reviewed-on: https://go-review.googlesource.com/c/160018
Reviewed-by: Russ Cox <rsc@golang.org>

gopherbot pushed a commit that referenced this issue Feb 11, 2019

go/constant: accept new Go2 number literals
This CL introduces go/constant support for the new binary and octal integer
literals, hexadecimal floats, and digit separators for all number literals.

R=Go1.13

Updates #12711.
Updates #19308.
Updates #28493.
Updates #29008.

Change-Id: I7a55f91b8b6373ae6d98ba923b626d33c5552946
Reviewed-on: https://go-review.googlesource.com/c/160239
Reviewed-by: Russ Cox <rsc@golang.org>

gopherbot pushed a commit that referenced this issue Feb 11, 2019

go/types: add tests for new Go 2 number literals
This CL ensures that go/types can now handle the new
Go 2 number literals. The relevant changes enabling
them in go/types were made in go/constant in the CL
https://golang.org/cl/160239.

R=Go1.13

Updates #12711.
Updates #19308.
Updates #28493.
Updates #29008.

Change-Id: I45c1387198fac94769ac59c5301d86b4e1a1ff98
Reviewed-on: https://go-review.googlesource.com/c/160240
Reviewed-by: Russ Cox <rsc@golang.org>

gopherbot pushed a commit that referenced this issue Feb 11, 2019

cmd/gofmt: normalize number prefixes and exponents
Rewrite non-decimal number prefixes to always use a lower-case base
("0X" -> "0x", etc.), and rewrite exponents to use a lower-case 'e'
or 'p'. Leave hexadecimal digits and 0-octals alone.

Comparing the best time of 3 runs of `time go test -run All` with
the time for a gofmt that doesn't do the rewrite shows no increase
in runtime for this bulk gofmt application (in fact on my machine
I see a small decline, probably due to cache effects).

R=Go1.13

Updates #12711.
Updates #19308.
Updates #29008.

Change-Id: I9c6ebed2ffa0a6a001c59412a73382090955f5a9
Reviewed-on: https://go-review.googlesource.com/c/160184
Reviewed-by: Ian Lance Taylor <iant@golang.org>

gopherbot pushed a commit that referenced this issue Feb 11, 2019

text/scanner: accept new Go2 number literals
This CL introduces text/scanner support for the new binary and octal integer
literals, hexadecimal floats, and digit separators for all number literals.
The new code is closely mirroring the respective code for number literals in
cmd/compile/internal/syntax/scanner.go.

Uniformly use the term "invalid" rather than "illegal" in error messages
to match the respective error messages in the other scanners directly.

R=Go1.13

Updates #12711.
Updates #19308.
Updates #28493.
Updates #29008.

Change-Id: I2f291de13ba5afc0e530cd8326e6bf4c3858ebac
Reviewed-on: https://go-review.googlesource.com/c/161199
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>

gopherbot pushed a commit that referenced this issue Feb 12, 2019

strconv: add 0b, 0o integer prefixes in ParseInt, ParseUint
This CL modifies ParseInt and ParseUint to recognize
0b and 0o as binary and octal base prefixes when base == 0.

See golang.org/design/19308-number-literals for background.

For #19308.
For #12711.

Change-Id: I8efe067f415aa517bdefbff7e230d3fa1694d530
Reviewed-on: https://go-review.googlesource.com/c/160244
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>
Reviewed-by: Rob Pike <r@golang.org>

nebulabox added a commit to nebulabox/go that referenced this issue Feb 18, 2019

cmd/compile: accept new Go2 number literals
This CL introduces compiler support for the new binary and octal integer
literals, hexadecimal floats, and digit separators for all number literals.

The new Go 2 number literal scanner accepts the following liberal format:

number   = [ prefix ] digits [ "." digits ] [ exponent ] [ "i" ] .
prefix   = "0" [ "b" |"B" | "o" | "O" | "x" | "X" ] .
digits   = { digit | "_" } .
exponent = ( "e" | "E" | "p" | "P" ) [ "+" | "-" ] digits .

If the number starts with "0x" or "0X", digit is any hexadecimal digit;
otherwise, digit is any decimal digit. If the accepted number is not valid,
errors are reported accordingly.

See the new test cases in scanner_test.go for a selection of valid and
invalid numbers and the respective error messages.

R=Go1.13

Updates golang#12711.
Updates golang#19308.
Updates golang#28493.
Updates golang#29008.

Change-Id: Ic8febc7bd4dc5186b16a8c8897691e81125cf0ca
Reviewed-on: https://go-review.googlesource.com/c/157677
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>

nebulabox added a commit to nebulabox/go that referenced this issue Feb 18, 2019

go/scanner: accept new Go2 number literals
This CL introduces go/scanner support for the new binary and octal integer
literals, hexadecimal floats, and digit separators for all number literals.
The new code is closely mirroring the respective code for number literals in
cmd/compile/internal/syntax/scanner.go.

R=Go1.13

Updates golang#12711.
Updates golang#19308.
Updates golang#28493.
Updates golang#29008.

Change-Id: I5315c6aaa7cfc41a618296be20e3acd5114d6b3c
Reviewed-on: https://go-review.googlesource.com/c/159997
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>

nebulabox added a commit to nebulabox/go that referenced this issue Feb 18, 2019

cmd/gofmt: test that Go 2 number literals can be formatted
R=Go1.13

Updates golang#12711.
Updates golang#19308.
Updates golang#28493.
Updates golang#29008.

Change-Id: Icd25aa7f6e18ed671ea6cf2b1b292899daf4b1a5
Reviewed-on: https://go-review.googlesource.com/c/160018
Reviewed-by: Russ Cox <rsc@golang.org>

nebulabox added a commit to nebulabox/go that referenced this issue Feb 18, 2019

go/constant: accept new Go2 number literals
This CL introduces go/constant support for the new binary and octal integer
literals, hexadecimal floats, and digit separators for all number literals.

R=Go1.13

Updates golang#12711.
Updates golang#19308.
Updates golang#28493.
Updates golang#29008.

Change-Id: I7a55f91b8b6373ae6d98ba923b626d33c5552946
Reviewed-on: https://go-review.googlesource.com/c/160239
Reviewed-by: Russ Cox <rsc@golang.org>

nebulabox added a commit to nebulabox/go that referenced this issue Feb 18, 2019

go/types: add tests for new Go 2 number literals
This CL ensures that go/types can now handle the new
Go 2 number literals. The relevant changes enabling
them in go/types were made in go/constant in the CL
https://golang.org/cl/160239.

R=Go1.13

Updates golang#12711.
Updates golang#19308.
Updates golang#28493.
Updates golang#29008.

Change-Id: I45c1387198fac94769ac59c5301d86b4e1a1ff98
Reviewed-on: https://go-review.googlesource.com/c/160240
Reviewed-by: Russ Cox <rsc@golang.org>

nebulabox added a commit to nebulabox/go that referenced this issue Feb 18, 2019

cmd/gofmt: normalize number prefixes and exponents
Rewrite non-decimal number prefixes to always use a lower-case base
("0X" -> "0x", etc.), and rewrite exponents to use a lower-case 'e'
or 'p'. Leave hexadecimal digits and 0-octals alone.

Comparing the best time of 3 runs of `time go test -run All` with
the time for a gofmt that doesn't do the rewrite shows no increase
in runtime for this bulk gofmt application (in fact on my machine
I see a small decline, probably due to cache effects).

R=Go1.13

Updates golang#12711.
Updates golang#19308.
Updates golang#29008.

Change-Id: I9c6ebed2ffa0a6a001c59412a73382090955f5a9
Reviewed-on: https://go-review.googlesource.com/c/160184
Reviewed-by: Ian Lance Taylor <iant@golang.org>

nebulabox added a commit to nebulabox/go that referenced this issue Feb 18, 2019

text/scanner: accept new Go2 number literals
This CL introduces text/scanner support for the new binary and octal integer
literals, hexadecimal floats, and digit separators for all number literals.
The new code is closely mirroring the respective code for number literals in
cmd/compile/internal/syntax/scanner.go.

Uniformly use the term "invalid" rather than "illegal" in error messages
to match the respective error messages in the other scanners directly.

R=Go1.13

Updates golang#12711.
Updates golang#19308.
Updates golang#28493.
Updates golang#29008.

Change-Id: I2f291de13ba5afc0e530cd8326e6bf4c3858ebac
Reviewed-on: https://go-review.googlesource.com/c/161199
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>

nebulabox added a commit to nebulabox/go that referenced this issue Feb 18, 2019

strconv: add 0b, 0o integer prefixes in ParseInt, ParseUint
This CL modifies ParseInt and ParseUint to recognize
0b and 0o as binary and octal base prefixes when base == 0.

See golang.org/design/19308-number-literals for background.

For golang#19308.
For golang#12711.

Change-Id: I8efe067f415aa517bdefbff7e230d3fa1694d530
Reviewed-on: https://go-review.googlesource.com/c/160244
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>
Reviewed-by: Rob Pike <r@golang.org>

nebulabox added a commit to nebulabox/go that referenced this issue Feb 20, 2019

cmd/compile: accept new Go2 number literals
This CL introduces compiler support for the new binary and octal integer
literals, hexadecimal floats, and digit separators for all number literals.

The new Go 2 number literal scanner accepts the following liberal format:

number   = [ prefix ] digits [ "." digits ] [ exponent ] [ "i" ] .
prefix   = "0" [ "b" |"B" | "o" | "O" | "x" | "X" ] .
digits   = { digit | "_" } .
exponent = ( "e" | "E" | "p" | "P" ) [ "+" | "-" ] digits .

If the number starts with "0x" or "0X", digit is any hexadecimal digit;
otherwise, digit is any decimal digit. If the accepted number is not valid,
errors are reported accordingly.

See the new test cases in scanner_test.go for a selection of valid and
invalid numbers and the respective error messages.

R=Go1.13

Updates golang#12711.
Updates golang#19308.
Updates golang#28493.
Updates golang#29008.

Change-Id: Ic8febc7bd4dc5186b16a8c8897691e81125cf0ca
Reviewed-on: https://go-review.googlesource.com/c/157677
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>

nebulabox added a commit to nebulabox/go that referenced this issue Feb 20, 2019

go/scanner: accept new Go2 number literals
This CL introduces go/scanner support for the new binary and octal integer
literals, hexadecimal floats, and digit separators for all number literals.
The new code is closely mirroring the respective code for number literals in
cmd/compile/internal/syntax/scanner.go.

R=Go1.13

Updates golang#12711.
Updates golang#19308.
Updates golang#28493.
Updates golang#29008.

Change-Id: I5315c6aaa7cfc41a618296be20e3acd5114d6b3c
Reviewed-on: https://go-review.googlesource.com/c/159997
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>

nebulabox added a commit to nebulabox/go that referenced this issue Feb 20, 2019

cmd/gofmt: test that Go 2 number literals can be formatted
R=Go1.13

Updates golang#12711.
Updates golang#19308.
Updates golang#28493.
Updates golang#29008.

Change-Id: Icd25aa7f6e18ed671ea6cf2b1b292899daf4b1a5
Reviewed-on: https://go-review.googlesource.com/c/160018
Reviewed-by: Russ Cox <rsc@golang.org>

nebulabox added a commit to nebulabox/go that referenced this issue Feb 20, 2019

go/constant: accept new Go2 number literals
This CL introduces go/constant support for the new binary and octal integer
literals, hexadecimal floats, and digit separators for all number literals.

R=Go1.13

Updates golang#12711.
Updates golang#19308.
Updates golang#28493.
Updates golang#29008.

Change-Id: I7a55f91b8b6373ae6d98ba923b626d33c5552946
Reviewed-on: https://go-review.googlesource.com/c/160239
Reviewed-by: Russ Cox <rsc@golang.org>

nebulabox added a commit to nebulabox/go that referenced this issue Feb 20, 2019

go/types: add tests for new Go 2 number literals
This CL ensures that go/types can now handle the new
Go 2 number literals. The relevant changes enabling
them in go/types were made in go/constant in the CL
https://golang.org/cl/160239.

R=Go1.13

Updates golang#12711.
Updates golang#19308.
Updates golang#28493.
Updates golang#29008.

Change-Id: I45c1387198fac94769ac59c5301d86b4e1a1ff98
Reviewed-on: https://go-review.googlesource.com/c/160240
Reviewed-by: Russ Cox <rsc@golang.org>

nebulabox added a commit to nebulabox/go that referenced this issue Feb 20, 2019

cmd/gofmt: normalize number prefixes and exponents
Rewrite non-decimal number prefixes to always use a lower-case base
("0X" -> "0x", etc.), and rewrite exponents to use a lower-case 'e'
or 'p'. Leave hexadecimal digits and 0-octals alone.

Comparing the best time of 3 runs of `time go test -run All` with
the time for a gofmt that doesn't do the rewrite shows no increase
in runtime for this bulk gofmt application (in fact on my machine
I see a small decline, probably due to cache effects).

R=Go1.13

Updates golang#12711.
Updates golang#19308.
Updates golang#29008.

Change-Id: I9c6ebed2ffa0a6a001c59412a73382090955f5a9
Reviewed-on: https://go-review.googlesource.com/c/160184
Reviewed-by: Ian Lance Taylor <iant@golang.org>

nebulabox added a commit to nebulabox/go that referenced this issue Feb 20, 2019

text/scanner: accept new Go2 number literals
This CL introduces text/scanner support for the new binary and octal integer
literals, hexadecimal floats, and digit separators for all number literals.
The new code is closely mirroring the respective code for number literals in
cmd/compile/internal/syntax/scanner.go.

Uniformly use the term "invalid" rather than "illegal" in error messages
to match the respective error messages in the other scanners directly.

R=Go1.13

Updates golang#12711.
Updates golang#19308.
Updates golang#28493.
Updates golang#29008.

Change-Id: I2f291de13ba5afc0e530cd8326e6bf4c3858ebac
Reviewed-on: https://go-review.googlesource.com/c/161199
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>

nebulabox added a commit to nebulabox/go that referenced this issue Feb 20, 2019

strconv: add 0b, 0o integer prefixes in ParseInt, ParseUint
This CL modifies ParseInt and ParseUint to recognize
0b and 0o as binary and octal base prefixes when base == 0.

See golang.org/design/19308-number-literals for background.

For golang#19308.
For golang#12711.

Change-Id: I8efe067f415aa517bdefbff7e230d3fa1694d530
Reviewed-on: https://go-review.googlesource.com/c/160244
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 20, 2019

Change https://golang.org/cl/163079 mentions this issue: text/scanner: don't liberally consume (invalid) floats or underbars

gopherbot pushed a commit that referenced this issue Feb 20, 2019

text/scanner: don't liberally consume (invalid) floats or underbars
This is a follow-up on https://golang.org/cl/161199 which introduced
the new Go 2 number literals to text/scanner.

That change introduced a bug by allowing decimal and hexadecimal floats
to be consumed even if the scanner was not configured to accept floats.

This CL changes the code to not consume a radix dot '.' or exponent
unless the scanner is configured to accept floats.

This CL also introduces a new mode "AllowNumberbars" which controls
whether underbars '_' are permitted as digit separators in numbers
or not.

There is a possibility that we may need to refine text/scanner
further (e.g., the Float mode now includes hexadecimal floats
which it didn't recognize before). We're very early in the cycle,
so let's see how it goes.

RELNOTE=yes

Updates #12711.
Updates #19308.
Updates #28493.
Updates #29008.

Fixes #30320.

Change-Id: I6481d314f0384e09ef6803ffad38dc529b1e89a3
Reviewed-on: https://go-review.googlesource.com/c/163079
Reviewed-by: Ian Lance Taylor <iant@golang.org>

nebulabox added a commit to nebulabox/go that referenced this issue Feb 21, 2019

text/scanner: don't liberally consume (invalid) floats or underbars
This is a follow-up on https://golang.org/cl/161199 which introduced
the new Go 2 number literals to text/scanner.

That change introduced a bug by allowing decimal and hexadecimal floats
to be consumed even if the scanner was not configured to accept floats.

This CL changes the code to not consume a radix dot '.' or exponent
unless the scanner is configured to accept floats.

This CL also introduces a new mode "AllowNumberbars" which controls
whether underbars '_' are permitted as digit separators in numbers
or not.

There is a possibility that we may need to refine text/scanner
further (e.g., the Float mode now includes hexadecimal floats
which it didn't recognize before). We're very early in the cycle,
so let's see how it goes.

RELNOTE=yes

Updates golang#12711.
Updates golang#19308.
Updates golang#28493.
Updates golang#29008.

Fixes golang#30320.

Change-Id: I6481d314f0384e09ef6803ffad38dc529b1e89a3
Reviewed-on: https://go-review.googlesource.com/c/163079
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment