Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: Go 2: improvements to raw strings #32590

Open
deanveloper opened this issue Jun 13, 2019 · 75 comments
Open

proposal: Go 2: improvements to raw strings #32590

deanveloper opened this issue Jun 13, 2019 · 75 comments

Comments

@deanveloper
Copy link

@deanveloper deanveloper commented Jun 13, 2019

Background

This proposal was branched off of #32190, which was a proposed HEREDOC syntax for Go. It was concluded that HEREDOC was not the correct syntax for Go to use, however the proposal did point out a large problem that Go currently has:

Problem

There is only one option to use raw strings, which is the backtick. The nature of how raw strings works means that raw strings themselves cannot contain backticks, meaning that the current workaround for including a backtick in a raw string is:

var str := `My backtick is `+"`"+` hard to use`

Raw strings are often used for storing large strings, such as strings containing other languages, or Go code itself. In many languages, the backtick has significant meaning. For instance:

  1. SQL uses backticks to signify a string that represents an identifier, such as a database name or table name. While they are not required for all identifiers, they are required if the identifier contains invalid characters (spaces, commas, etc) or if the identifier matches names with a keyword. It also seems to be good practice in general to surround database and table names in backticks.
  • Example: SELECT * FROM `database`.`table`
  1. Kotlin uses backticks in a similar fashion.
  • Example: fun `a method with spaces`() { ... }
  1. JavaScript uses backticks to indicate format strings, which allow people to embed expressions inside of their strings.
  • Example: let str = `HELLO ${name.toLocaleUpperCase()}`

Of course there are far more examples of languages where the backtick is a significant character in the language. This makes embedding these languages in Go very hard.

Proposed Solution

If there were a fixed number of ways to declare raw strings, the problem would, no matter what, arise that you would be unable to put Go code inside of Go code without some kind of need to transform the code. This means that there needs to be a variable way to create raw strings.

This proposal highlights one brought up here. It essentially improves on the current way to declare raw strings, allowing the following syntax:

var stmt = SQL`
SELECT `foo` FROM `bar` WHERE `baz` = "qux"
`SQL

var old = `
this, of course, still works
`

var new = 고`this
also works
    you can also use 고 AND `backticks` (separately) in the string!
`고

Essentially, raw strings can be prefixed with a delimiter, and the string is then terminated with a backtick followed by the same delimeter.

Strings which are densely populated with words and backticks may make it hard to pick a word to use as the delimiter for the raw string, as the word may appear inside the string, which would end the string early and cause a syntax error. Allowing any identifier to be used as a delimiter would allow non-ascii characters to be used as well, meaning that in special cases, when it's really needed, one can use a non-ascii character as their delimiter.

Concerns

@jimmyfrasche #32190 (comment)

Implementation-wise, the problem with user-specified delimiters is that they have to be handled during lexing, which adds complexity to a simple, though still somewhat involved, stage and would need a lot of explanation in the language spec.

I don't like the idea of complicating the language. I do not work with the internals of the language, so I am unsure of the magnitude of complication to the lexer that this change would bring. If it is too much, I don't think that it would at all be worth it, and maybe one of the alternatives below would be a better fit.

@ianlancetaylor #32190 (comment)

My only concern with [syntax] is that it doesn't lead with the fact that it is a string. C++ (R"delim( string )delim")) and Rust (r#" string "#) and Swift (#" string "#) are more clear as to when a string is starting.

I share this sentiment. My response to this here was that establishing a convention to use short, noticable identifiers (ie RAW, JS, SQL, etc) help with noticing where the string starts and ends. This could (possibly) be enforced by golint, but I'm not sure if that is a good idea or not.

Other Alternatives brought up

In #32190, there were several other alternatives that tried to achieve the same goal:

Variable numbers of backticks

Essentially, you could start the raw string with a certain number of backticks, and it would have to end with the same number of backticks.

    `````
    A raw string which can contain up to 4 ```` backticks in a row inside of it
    `````

This solution still had problems though. Strings cannot start with an even number of backticks, because any even number of backticks could also be interpreted as an empty string, introducing ambiguities. It also causes developers a bit of fuss when trying to get it to work inside of markdown, as markdown uses multiple backticks in order to signify a block of code.

Also, the strings could not start or end with backticks, which would be an unfortunate consequence.

Variable number of backticks + no empty raw strings

This one is a breaking change, however I think it is my favorite solution out of all of the alternatives. It's the exact same as the previous one, but Go also introduces a breaking change to disallow empty raw strings. There is no need for raw strings to be used to represent an empty string, since the normal "" can do that, and is much more preferable. The only code this would break is people who have used a raw string to define an empty string by doing something like x := `` or funcCall(``, ...). It may be good to do some research on if empty raw strings are ever used in real code.

This solution still has the issue of being annoying to use with markdown's code fences. The argument was used that we shouldn't make language decisions based on other languages, however I personally do not like this argument. Sharing code is part of what a programmer does, and Markdown is a very widely used markup language that uses multiple backticks in a row to define a code fence. This feature may make it a bit difficult to share Go code over anything that uses Markdown (slack, github, discord, and other services).

Despite making it difficult to share code via markdown-enabled chats, it is still easy to share code via something like gist.github.com or play.golang.org. If my original proposal proves to not work very well (doesn't feel Go-like, too difficult to implement, etc) I would love for this solution to be accepted in place.

Variable number of backticks + a quote

This proposal is actually pretty nice. It's similar to the previous proposal. Essentially, the starting is N backticks (N >= 2) followed by a quotation mark, and the ending delimiter is a quotation mark followed by the same number of backticks. Example:

s := ``"this is a `raw` "string" literal"``
fmt.Print(s)

// prints:
// this is a `raw` "string" literal

This syntax is actually very nice in my opinion. It fixes the "odd-number-only" ambiguity from the previous example, as well as fixing the Markdown issue (as code fences must occur on their own line). It also fixes the "strings starting/ending with backticks" issue.

The only issue with this syntax is that it doesn't seem to work well with existing raw strings. I don't personally have data about how often this occurs, but I'd imagine that there are several times where raw strings are used to describe strings with quotes in them, making code like x := `"this is a string"` common. Newcomers to Go may see this and think that the `" is the delimiter to the raw string, when in reality the ` is the delimiter and the " is part of the string.

However that critique may be a bit nitpicky. I do like this syntax a lot.

Choosing a symbol pair that nobody uses

This alternative stated that Go should add another symbol to use to declare raw strings in Go. For instance, to start the string and to end the string. Go code is defined to be UTF8 so file formatting issues should not happen. Another proposed idea was (U+2261 IDENTICAL TO).

This solution also has problems. What if our string has both backticks AND strange symbols (for instance if you were defining a list of mathematical symbols)? Or, what if you were trying to embed Go syntax inside of your strings? Also, the symbol is hard to type and not easy to find, so it may not be a good fit as a string delimiter.

Other languages

  1. C++ R"delim(string)delim"
    • In my opinion, I personally hate the asymmetry of prefix-strings, they look sloppy to me and seem too much like they were trying to hack in features, so I don't really like this solution.
  2. Rust r#"string"#
    • Same issue that I had with C++: the asymmetry and "hackiness" of prefix-strings ruins it for me. Also, a fixed number of ways to define a string means that if one wants to put a Go raw string inside of a string (ie pattern matching for code generation), they will run into issues.
  3. Swift #"string"#
    • Again, a fixed number of ways to define a string means it's hard to pattern-match Go raw strings for code generation.

It's important that we have some kind of variable delimiter, as that way if the string we are embedding somehow contains it, it is easy to change the string's delimiter in order to avoid the issue.

The delimiter doesn't have to be an identifier like it is in this main proposal, it could also be varying the number of backticks like the one a few paragraphs up.

Conclusion

Raw strings in Go are often used to be able to copy-paste text to be used as strings, or to embed code from other languages (such as JS, SQL, or even Go) into Go. However, if that text contains backticks, we need some way to make sure that those backticks do not terminate the string early.

I believe that the way to do this is allowing an identifier to precede the string, and to make sure that the terminating backtick must be followed by the same identifier in order to terminate the string.

var markdown = MD`
### Thank you for reading :)
`MD
@gopherbot gopherbot added this to the Proposal milestone Jun 13, 2019
@gopherbot gopherbot added the Proposal label Jun 13, 2019
@deanveloper
Copy link
Author

@deanveloper deanveloper commented Jun 13, 2019

Just found this as well: #24475

I did not see this one while making this proposal, my bad. But it does contain another interesting suggestion by @bcmills #24475 (comment)

Specifically, we could treat as a raw string any sequence of characters which:

  • begins with a QUOTATION MARK character in Unicode catogory Pi, and
  • ends with the corresponding QUOTATION MARK character in Unicode category Pf.
@davecheney
Copy link
Contributor

@davecheney davecheney commented Jun 13, 2019

SQL uses backticks to signify a string that represents an identifier, such as a database name or table name.

I do not believe this is correct. Certainly for databases like MySQL and Postgress their quoting character is ', ASCII 0x27.

Ref: https://dev.mysql.com/doc/refman/8.0/en/string-literals.html Section 9.1.1
Ref: https://www.postgresql.org/docs/11/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS Section 4.1.2.1

@cespare
Copy link
Contributor

@cespare cespare commented Jun 13, 2019

@davecheney these databases use single quotes for strings, yes. (Some allow " as well.)

Backticks are used by MySQL (and sqlite? any others?) if you need to quote identifiers: https://dev.mysql.com/doc/refman/8.0/en/identifiers.html (search for "backtick").

(Postgres uses " for this purpose.)

@cespare
Copy link
Contributor

@cespare cespare commented Jun 13, 2019

I like @deanveloper's main proposal, but on reflection I like @ianlancetaylor's idea (``" a `raw` "string" "``) even more. It is less flexible than using an arbitrary identifier, but that extra power seems largely pointless and introduces another place where people will have to choose/argue about a name.

@davecheney
Copy link
Contributor

@davecheney davecheney commented Jun 13, 2019

@cespare Thank you for the correction.

I continue to assert that back ticks for SQL string construction is not a valid argument for a language change proposal -- users should not be constructing SQL query strings by hand. The fact that

a. Alternative syntax like ' (which I believe is part of the SQL standard, but I couldn't find a linkable reference) is supported
b. You only need to use back tick in cases where you have to quote reserved words, and the alternative exists now +""+`. Its not pretty, but that's because people shouldn't be constructing SQL strings by hand.

Gives weight to this position.

@deanveloper
Copy link
Author

@deanveloper deanveloper commented Jun 13, 2019

@davecheney I gave other examples, such as JavaScript and Go, which are both languages that I can see being put into raw strings. JavaScript, of course because Go is used a lot for web backends, so, depending on the circumstance, it could make more sense to embed a small script into a raw string rather than create an entire separate file for it. Go may make sense for code generation purposes if you wish to detect a raw string in a file.

Also, there is the possibility that someone may just need to encode a string that has many backticks in it.

I still personally believe that SQL is a valid use-case for this. SQL Injection attacks are caused by people using string concatenation in SQL queries, not necessarily because they are crafting queries by hand. The database/sql library actually takes care of sql injection with the ? syntax, which allows people to pass in user-generated arguments with automatic sanitation of inputs. It's a pretty well-known feature as far as I'm aware. Asking all users to use third-party libraries in order to query tables is unreasonable IMO, especially when Go already comes with standard library support for database/sql.

a. Alternative syntax like ' (which I believe is part of the SQL standard, but I couldn't find a linkable reference) is supported

Double quotes are technically supported by MySQL, but only if ANSI_QUOTES mode is enabled (as shown by the page linked by @cespare). Enabling this mode also requires that string literals no longer use double quotes.

Also, while backticks are not technically required in SQL languages, they are still common practice. If a DBA sends me a complex query to use, I don't want to go through the whole thing and remove each backtick, or need to put in a `+"`"+` instead.

@MOZGIII
Copy link

@MOZGIII MOZGIII commented Jun 13, 2019

@davecheney, backticks are not in the SQL standard, they're database specific. For example MSSQL doesn't use backticks, but MySQL and Postgres do. With those DBs there's no alternative to using backticks sometimes.
Often you just need to encode an SQL string literal without any dynamic construction (i.e. no variable substitution or conditional logic). This is not considered "constructing SQL by hand" and is, on the contrary, very recommended, while usage of query builders and ORMs is somewhat discouraged in Go.

@davecheney
Copy link
Contributor

@davecheney davecheney commented Jun 13, 2019

while usage of query builders and ORMs is somewhat discouraged in Go.

Could you please cite a source for this position. Thank you

@davecheney
Copy link
Contributor

@davecheney davecheney commented Jun 13, 2019

To clarify my position. If you’re using sql you should be using prepared statements. To the best of my knowledge when that is done the dB drivers take care of quoting for identifies and values. This is why I assert sql queries are not a good premise for this proposal.

@deanveloper
Copy link
Author

@deanveloper deanveloper commented Jun 13, 2019

Could you please cite a source for this position. Thank you

While this is not a direct source (I don't have one, but I also am not the one who claimed this fact so I think I get a bit of leeway):

The lack of support for generics is a huge bummer for SQL Builders, because they cannot have type safety, which is really the only advantage they had over ? syntax. They also typically heavily rely on chaining syntax which Go also discourages with it's semicolon placement rules. This was also referenced in the design doc for error handling here (scroll to the ? operator question). While not stating that chaining is discouraged, definitely stating that chaining is uncommon. Again, this is because of semicolon placement rules, which cause chaining (aka builder syntax) to become a bit ugly:

var builder myBuilder
x := builder.
    AddThis().
    AddThat().
    SetFoo(bar).
    Baz().
    Build()

This is not a problem for simple queries, however as queries get more complex, it gets exponentially worse.

To clarify my position. If you’re using sql you should be using prepared statements. To the best of my knowledge when that is done the dB drivers take care of quoting for identifies and values. This is why I assert sql queries are not a good premise for this proposal.

Literals are common in SQL queries, in which case I need to worry about quoting. I don't want to replace every single one of my literals with a ?. For instance, a simple query for selecting every user who used a certain coupon:

SELECT `user` FROM `users` JOIN `transactions` USING (`uid`) WHERE coupon_applied = "someCoupon"

There are definitely times where I want string literals embedded in my queries, and again as queries get more complex I definitely don't want to be moving string literals into ?.

@MOZGIII
Copy link

@MOZGIII MOZGIII commented Jun 13, 2019

Well, the way Go standard library (database/sql) is implemented is clearly in favor of writing the SQL literals. The reason for that is if you use some form of SQL builder or ORM you often don't pay enough attention to what's going on under the hood. This, in turn, leads to nasty performance and code logic issues.

Prepared or not, you still need to communicate your statement (aka query) to the database server, and this usually requires writing your own SQL literal string. Sure it'd be a big mistake to use string concatenation instead of binding the query params. Binding works for both usual and prepared statements. Btw, prepared statements have their own issues - for example they're very hard to manage with transaction based connection pooling (i.e. via pgbouncer) - so I tend to not use them that much. Performance wise, in many typical cases prepared statement slow down the overall system performance, rather than speed it up - so use them with caution and measure the effect.

@davecheney
Copy link
Contributor

@davecheney davecheney commented Jun 13, 2019

@MOZGIII i disagree with most of your advice, yes prepared statements have a cost, but a very clear benefit, and you don't need to quote the ?'s which is my point.

There must be a better use case for this proposal than SQL strings.

@deanveloper
Copy link
Author

@deanveloper deanveloper commented Jun 13, 2019

@davecheney I've mentioned both in the original proposal and my first reply to you other possible use cases, but you never seem to address them. Even if they're relatively minor, there's just no good way to encode a multiline string that contains backticks. Is that really too much to ask, especially out of a general-purpose programming language?

@davecheney
Copy link
Contributor

@davecheney davecheney commented Jun 13, 2019

@deanveloper if we take SQL quoting out of the proposal, then the use cases you're suggesting are writing snippets of other languages in Go string literals? How common is this in the wild?

@MOZGIII
Copy link

@MOZGIII MOZGIII commented Jun 13, 2019

@davecheney When variable binding is used, the actual variables and the query are typically passed as separate units to the database engine. I can assure that it is the case for postgres, but less advanced databases actually may do something different. But nonetheless, in postgres, the data packet to the database over the wire can be represented as the following tuple: (string, Value[]). If you use the SQL literal SELECT ?::text it will be sent over the wire as is (pretty much). Along with it, the bound variable will be sent, for example "hello world". Moreover, the bound value will be encoded with the proper wire format for strings. On the server side, the database' will first run the received statement through the parser, then match the ?s to the values and then execute the actual query. Nowhere in this sequence the value is quoted or put in place of the ? in the actual query. This is what we call variable binding. One of the additional bonuses of this approach is that database parser layer typically caches the parsing for statements (the statements tend to repeat a lot with this approach) - and it boosts the performance quite significantly.

Now, prepared statements. They can also use variable binding, but they don't have to, the same way as the regular, non-prepared statements. The problem with prepared statements is it's quite difficult to manage them on a per-session level, and nobody does that. The alternative is preparing the statement for every operation independently. This is slow, because it often causes multiple round trips. This may become a critical bottleneck, as it easily doubles the execution time in a well optimized system. As I said, variable binding is not only possible with the prepared statements. Under the hood it is implemented even for a regular SQL queries: https://github.com/jackc/pgx/blob/762e68533f0090ecb6bb1166d51966b326597ec7/query.go#L410-L453
This code completes in only one round trip, as long as all value types (OIDs) are known upfront.

Anyhow, let's get back to literals - I don't want this to be an SQL discussion. I would say that the lack of support to just copy and paste arbitrary string (SQL with quotes in particular) is a design flaw of the existing implementation. Especially while iterating on the whatever thing you need to represent as a literal - it's much easier to set the boundaries once rather than escaping the backticks on every iteration.

@MOZGIII
Copy link

@MOZGIII MOZGIII commented Jun 13, 2019

@davecheney Embedding markdown is another example. I feel like pretty much every other language that has backticks as part of the syntax may cause issues if we attempt using it in the current raw string literals implementation. Those are bash, ruby, perl, TeX and many more. You never know all the ways the feature that's not in the language could've been used. Frankly, even for features that are in the language it's very hard to tell all possible uses.

I think we should not accept "SQL is not a good example" as an argument. The rationale is a) the lack of a better example doesn't prove there's no problem, and b) SQL is not just a practical example, it is an actual pain point that I've encountered while solving my day-to-day tasks with Go.
Surely, more examples can shape the way we address the problem, but, unless we address the SQL pain point separately (which I don't see proposed here), I don't see why would we want to ignore it.

@deanveloper
Copy link
Author

@deanveloper deanveloper commented Jun 13, 2019

@davecheney I'm going to come at this from a different angle, actually. Regardless of if you want to use Go for code generation, which I personally do quite a bit, raw strings the way they are now are just poorly designed.

Raw strings are designed to store large amounts of text, or text which contains special characters (" and \), or even both, into a variable. However, if the string contains a different special character, then the entire purpose behind raw strings falls apart. At least in other languages they have safeguards such as using two different characters in a row as a delimiter to a string, making it quite a bit harder for the string to be terminated early. However, even this solution still has the problem with code generators being a bit messier.

@alanfo
Copy link

@alanfo alanfo commented Jun 13, 2019

The more argument (and lack of agreement) I see on this matter, the more convinced I become that it's not worth doing anything here and we should just live with what we have.

The present syntax for raw strings has the desirable properties (for a simple language such as Go) of being very easy to type, parse, read, explain, understand and remember. The only problem it has is not being able to include the back-tick character and opinions differ on how acute this problem actually is in the wild.

All the proposed solutions (including those suggested by myself in #32190) either seriously compromise one or more of these properties and/or deal with the back-tick problem at the expense of creating a similar problem with some other (more rarely used) character.

It's also worth remembering that, apart from the workaround mentioned by @deanveloper in his opening post, there are (at least) two other ways of dealing with the back-tick problem at the present time:

  1. Changing the back-ticks in the raw string to some other unused character and then applying strings.Replace on the resulting string to change them back to back-ticks.

  2. Using fmt.Sprintf in the following manner:

    s := fmt.Sprintf(`This is a "raw" string including %cback-ticks%[1]c.`, '`')
    fmt.Println(s)
    // This is a "raw" string including `backticks`.

Admittedly none of these solutions is exactly "nice" but, on balance, I think they are better than introducing some new (and probably contentious) syntax to deal with the back-tick problem which I don't think everyone regards as being particularly serious in any case.

@MichaelTJones
Copy link
Contributor

@MichaelTJones MichaelTJones commented Jun 13, 2019

This has my support. I'm a voice for Unicode as an out of (ASCII) bounds signalling means, but admit that while it solves the easy case of quoting all normal situations, it cannot alone address recursive self quoting; for that we must have a case-specific delimiter, and identifier-backtick is a clean, simple, universal mechanism.

It is trivial to parse so I dispute any pushback on inconvenience of implementation. In addition to the likely uses...

`say, "plugh"`

xxx`say, "plugh"`xxx

 Δ`say, "plugh"`Δ

...it also allows an extreme case: generate a huge random integer (256 bits, say), encode that in identifier form (letter or underscore followed by (letter|underscore|digit)*), and use that to blind quote text without looking at it and knowing it will work. This may never come to pass, but I like that it could:

randomQuoteTag2f4d87ef38eb88c21a00104006339a15e9cbe43f2610c247314cef54ab5f5db8`unlikely to have collisions`randomQuoteTag2f4d87ef38eb88c21a00104006339a15e9cbe43f2610c247314cef54ab5f5db8

This is something you could rely on when, say, packaging files in strings, like a go generate tool that pastes a copy of a source file into that source file.

@deanveloper
Copy link
Author

@deanveloper deanveloper commented Jun 13, 2019

Admittedly none of these solutions is exactly "nice" but, on balance, I think they are better than introducing some new (and probably contentious) syntax to deal with the back-tick problem which I don't think everyone regards as being particularly serious in any case.

My personal issue with this solution is that you then must apply transformations to the string before the code can compile. This is especially undesirable if you do not have syntax highlighting, which is a crutch that Go should not depend on. It is also annoying to write a program to apply the string transformation for you, as you cannot put the original string into a Go program.

@kardianos
Copy link
Contributor

@kardianos kardianos commented Jun 13, 2019

@deanveloper I write a lot of SQL and SQL system and I write a lot of Go. I'm neutral on the core of this proposal. However the motivation for a change is important.

When building SQL strings, you may want to quote identifiers and quote text/time/(a few other cases) values. It is assumed that you will use value parameters where appropriate. Also of note, a prepared query and and a query that uses value parameters are not directly correlated in most system. Different database system support different ways to quote identifiers and values. SQL Server uses square brackets [] to quote identifiers and single quote marks for values only. PostgreSQL uses double quote marks for identifiers and single tick marks for values.

Let's take your SQL problem and address it first. For this problem, the solution you have will work, but is not one I would recommend. It is too ad-hoc for a system. Spend a half a day and we can make a better solution.

Spend a little bit more time in discovery to find other motivating pain points. The example you gave just isn't that motivating to me.


A better solution

package sqls

import (
    "github.com/golang-sql/sqlexp"
)

// Replacer sets up replacement classes for SQL strings.
type Replacer struct {
    Quoter sqlexp.Quoter // If nil, ANSI SQL is assumed.
    IdentiferReader rune // If zero value, ':' is used.
    VariableReader rune // If zero value, '@' is used.
}

// Replace reads a format string and replaces identifier names that start with the IdentifierReader
// followed by by at least one letter class rune and zero or more letter or number class runes.
// Similar for VariableReader names. Values may be either a map[string]interface{} or a typed
// struct. If a struct is used reflection will be used.
func (r Replacer) Replace(f string, values interface{}) (string, error) {
   //...
}

func TestReplacer(t *testing.T) {
    r := &Replacer{}
    out, err := r.Replace(`
select
    :MyColumns/t1,
    :OtherColumns/t2
from
    :MyFirstTable t1
    join :MySecondTable t2 on t1.ID = t2.:LinkColumn
where
    1=1
    and :Where1/t1 = @MyValue
`,
    map[string]interface{} {
        "MyColumns": []string{"ID", "Name"},
        "OtherColumns": []string{"BookName=Book"}, // Identifiers are inspected for "=" for alias.
        "MyFirstTable": "Author",
        "MySecondTable": "Book",
        "LinkColumn": "AuthorID",
        "Where1": "Location",
        "MyValue": "US",
    })

     fmt.Println(out)
     // OUTPUT
     /*
select
    t1."ID", t1."Name",
    t2."Book" as "BookName"
from
    "Author" t1
    join "Book" t2 on t1.ID = t2."AuthorID"
where
    1=1
    and t1."Location" = 'US'
*/
}

There are lots of ways to solve the problem you presented. This would be one of them. I would argue it would do a better job of what you are wanting then what raw strings (current or improved) can give you. You could implement this as a runtime step or as a pre-compile step. I've also used text/template + some custom template functions to construct SQL.

Constructing SQL is fine. Carefully including values in SQL text can be okay if done through a system (like above) to prevent silly mistakes. I don't care if you use parameters or not, I don't care if you prepare your SQL or not (modern SQL engines a prepare is useless).

@deanveloper
Copy link
Author

@deanveloper deanveloper commented Jun 13, 2019

Again, SQL is not the only target of this. Raw strings as they are now, are poorly designed as illustrated in #32590 (comment). That comment is a better illustration of the problem that we currently face than all of this arguing on SQL Queries and how they should be done.

@MOZGIII
Copy link

@MOZGIII MOZGIII commented Jun 13, 2019

@kardianos that, or there could've just been a string literal. It is a beautiful workaround for a problem that shouldn't exist in the first place. With this you can either do a compile-time codegen - in which case just having a .sql file wrapped with Go literals with proper escaping is sufficient, and the end result can be a statically allocates string from the binary image - or it can be applied at runtime - probably during the system initialization, cause actually constructing an SQL query like that for every operation would result in lost cycles (unless you actually generate different strings - which is possible, but comes close to what I'd avoid doing in the first place, as the dynamic values, either from the user or from the system itself, we always use the standard API to pass them to the driver via Args list).

The problem with the example above is that, while it solves one part of the problem - and that is allowing you to get a program that invokes the query you want - it doesn't address other parts - among those are the ability to author the same literal that the app will use. This is more important for literals talk than the ability to actually somehow make the string value that you want appear at runtime. Go already addresses this problem, it just does it poorly (we have the ` notation in the language).

As a counter example of why we actually need to have good literals for raw strings: any byte array (even non-unicode) we can represent in a Base64 encoding, embed into the regular " string (no need for even `), decode it in runtime and be happy with it. The downsides of this solution should be obvious.

@kardianos
Copy link
Contributor

@kardianos kardianos commented Jun 13, 2019

Spend a little bit more time in discovery to find other motivating pain points.

@kardianos
Copy link
Contributor

@kardianos kardianos commented Jun 13, 2019

@MOZGIII Any string literal by itself won't automatically escape value or identifiers, nor will it expand an array of columns nor will it escape and join together a list of strings or other values. I personally find these things important. You may not.

@MOZGIII
Copy link

@MOZGIII MOZGIII commented Jun 13, 2019

@kardianos so, what you mean is your code does something valuable that's not directly correlated to representing literals, and as a bonus it solves the issues with `. That's great, but what if you don't need all that other stuff? I personally sometimes just need to put whatever SQL I have agreed on with my peers in to the code - and with that workflow it's important that the query is actually the same that was used in prototyping so other people are familiar with it. Especially under the conditions where time is money and the SQL is typically 20+ lines. That said - those are all second order issues, and they key thing here is to improve the literals the language provides.

Are we still in the "why?", or are we in the "how? already? If we're in the "how?" then I don't see the point of looking into the workarounds to the SQL use case - let's use that example to see what literals we can offer to solve the pain point.

PS: just noticed markdown supports multiple backticks for verbatim substrings. This means we can play around with this idea easily!

Example: (working)

1) `a`
2) ``a``
3) ```a```
4) ````a````
5) `` ` ``
6) ``` ` ```
7) ```` ` ````

Result:

  1. a
  2. a
  3. a
  4. a
  5. `
  6. `
  7. `

Example: (not working)

1) ``
2) ```
3) `````
  1. ``
  2. 
    

I must say I like qwe`value`qwe-style the most currently. Something like qwe"value"qwe or ##`value`## works too, as well as ```"value"```. To choose one among all the possible options, I guess the best way is to try writing some code pretending the change is already there and compare the solutions.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jul 10, 2019

@MichaelTJones I don't hate it. But when reading the code I'd somewhat prefer to see immediately that I am looking at a string, rather than seeing iant, which looks like an identifier, and then seeing ", at which point I see that it is not an identifier, but is actually a string.

I think that most other languages have raw string literals that preserve that property: it's clearly a string from the start. (C++: R"delim( string )delim"; Rust: r#" string "#; Swift #" string "#).

@deanveloper
Copy link
Author

@deanveloper deanveloper commented Jul 10, 2019

@ianlancetaylor I can agree a bit with that, I think that @DeedleFake's suggestion may be a better syntax.

Empty raw strings are pretty useless and rarely used [citation needed], it might be a good idea to break a small amount of compatibility and disallow empty raw strings, and allow a variable number of backticks instead.

@MichaelTJones
Copy link
Contributor

@MichaelTJones MichaelTJones commented Jul 10, 2019

@agnivade
Copy link
Contributor

@agnivade agnivade commented Jul 11, 2019

I am sympathetic to @donatj's comment about pasting a raw string as I have encountered something very similar. In my case, it was editing an existing string rather than just copying it. I will just copy over my comment from #32190 (comment) again since this proposal is for general raw string improvement.

Comment copied below---

I had an html template as a raw string. Now in that template, I have <script> tags and javascript inside those. I was going to use a console.log() with the ${} notation for variables inside a <script> tag, and then immediately realized that I am stuck.

What would have been a nice

console.log(`time taken: ${performance.now()-start}`)

became

console.log(` + "`time taken: ${performance.now()-start}`);" +
`
@qiangli
Copy link

@qiangli qiangli commented Aug 23, 2019

I might be a little off track. How about something like this (https://github.com/jteeuwen/go-bindata) built into the language? It will solve all the probems here and what's more, a powerful feature for including static asset content into the code for many other uses.
Here is one approach:
The file names (any file not ending in ".go") in the same directory can be referenced as variables of []byte. the variable name is the file name without the extension (filepath.Ext()) it follows the same export convention also.
e.g.:

mypkg/mysql.sql
mypkg/Javascript.js
....
mypkg/xxx.go:


pakcage mypkg

const sqlQuery = string(mysql)
var script = string(Javascript)

"Javascript" is exported and can be imported and referenced as "mypkg.Javascript" in other packages.
build flag is required to turn on this feature where one can specify what file extesions are included/excluded.

A another less dramatic change is to introduce a standard package such as "text/bindata" and standard functions such as func Raw(name string) []byte.

@deanveloper
Copy link
Author

@deanveloper deanveloper commented Aug 23, 2019

That solution doesn't quite work since file names can contain many characters that don't fit into Go identifiers, such as spaces, quotation marks, etc. It would also cause confusion to many as the variable is not declared anywhere.

It is an interesting idea though, since it would result in a compile-time error if anything happened to the file.

(Also, just pointing out that your "text/bindata.Raw" looks just like "io/ioutil.ReadFile" 😉 )

Either way, I think that solution solution misses the point - raw strings in Go accomplish their goal poorly. They are made to capture exactly what is in between two delimiters, no different. Newlines, backslashes, quotation marks, ALL are contained in the string. Except backticks, which are a fixed delimiter.

This proposal aims to add some kind of variable delimiter (hopefully either a prefix/postfix identifier, or variable number of backticks) for raw strings in Go, which would assure that there is not a single possible string that isn't able to simply be pasted into a raw string without transformations.

@cespare
Copy link
Contributor

@cespare cespare commented Sep 26, 2019

Here is another real life situation that I don't believe has been mentioned yet.

The language defined by text/template uses " and ` for strings, just like Go, with the same semantics.

Inside a template, I want to write a string literal that includes many " characters. To avoid excessive escaping, I wrote the literal using a raw string with backticks. However, the template text is itself inside Go code, so this doesn't work. I ended up doing something like this:

var tmplText = `{{with $json := ` + "`" + `{"hello": "world"}` + "`" + `}}
{{$json}}
{{len $json}}
{{end}}
`

whereas with one of these proposals, I could just directly write the text I want to use. For example, using my favored syntax (@ianlancetaylor's):

var tmplText = ``"{{with $json := `{"hello": "world"}`}}
{{$json}}
{{len $json}}
{{end}}
"``
@integrii
Copy link

@integrii integrii commented Feb 9, 2020

Let's say I am writing some code that creates a database table. I created the table in MySQL workbench and want to put the exact SQL that workbench creates into my code. That code might look something like this:

CREATE TABLE IF NOT EXISTS `mydb`.`user` (
  `email` VARCHAR(255) NOT NULL,
  `hash` VARCHAR(64) CHARACTER SET 'DEFAULT' NOT NULL,
  `create_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `admin` TINYINT NOT NULL DEFAULT 0,
  `DisplayName` VARCHAR(255) NULL,
  PRIMARY KEY (`email`),
  UNIQUE INDEX `email_UNIQUE` (`email` ASC) VISIBLE)

Without some kind of unique EOF delimiter like the original proposal suggests, how would you do this?

  • Search/replace all backticks to quoted backticks with concatenation sourrounding them
  • Try to switch all backticks to single quotes or something and hope it still works
  • Load a file with the contents
  • convert it to base64 and decode it into a variable
  • ??? this is getting silly
@alanfo
Copy link

@alanfo alanfo commented Feb 9, 2020

Although I've been skeptical in previous posts about the need to do anything here, after staring at the latest slab of back-tick infested SQL I've concluded that we should try and come up with a simple solution as the current workarounds are just too painful.

Reading through the whole thread from the start, I think we may have lost sight of the original problem - allowing raw strings to contain back-ticks - and gotten bogged down with trying to find a solution which can deal with absolutely everything.

Suppose instead we introduce a new optional delimiter for raw strings namely triple double-quotes : """a `raw string` containing back-ticks.""". Apart from being able to include any number of back-ticks, the advantages of this would be:

  1. The delimiter is the same at both ends and therefore easy to parse.

  2. Its backwards compatible and people would probably prefer to continue using back-ticks as delimiters in the majority of cases as they're shorter to write.

  3. It is no problem if the raw string begins or ends with a single or double double-quote character.

  4. A number of languages use a triple double-quote as a raw string delimiter so this should be quite familiar to many developers and not therefore take much learning. These languages now include Java which is using them for its 'text block' feature which I linked to earlier. Give the size of the latter's code base you'd think they must be fairly confident that it's a good combination to use from a rarity point of view.

  5. The use of double quotes rather than some more esoteric delimiter meets @ianlancetaylor's point about it 'looking like a string'.

  6. The effect on the language spec should be minimal - it just needs to say that a raw string either begins and ends with a back-tick or with a triple double quote. The former can't include a back-tick but can include triple double-quotes. The latter can't include a triple double-quote but can include a back-tick so the two are complimentary and otherwise behave exactly the same.

The only disadvantage I can think of is that you wouldn't be able to include BOTH a back-tick and a triple double-quote in the same raw string. But how many times does that come up in practice? With the possible exception of language source code, very rarely indeed I'd suggest.

@MOZGIII
Copy link

@MOZGIII MOZGIII commented Feb 9, 2020

The only disadvantage I can think of is that you wouldn't be able to include BOTH a back-tick and a triple double-quote in the same raw string. But how many times does that come up in practice?

In my practice, it happened at least once, in a non-source-code context. 😄

But whatever the solution is - it's better than nothing. 😄 At this point, I'd just take it. Id' say, getting a more generic would be more preferable to me, but it can be deferred till after we solve the major pain point with SQL - and then we'll see if it still needs work.

@donatj
Copy link

@donatj donatj commented Feb 12, 2020

I still believe a custom delimiter solution is ideal. A very limited HEREDOC-esq syntax such that you define your own delimiter and close with it would be ideal.

I know, I know

It was concluded that HEREDOC was not the correct syntax for Go to use

A big chunk of this conclusion came down to complicating features HEREDOC brings to other languages. None of these are inherent to all HEREDOC implementations though, and none of them would need to be implemented here. We don't need to cargo-cult them in.

The solution can be simple. It doesn't have to bring in every stupid feature other HEREDOC's have.

So hear me out, and then I'll slip back into the abyss.

Here's my rough proposal:

image

It fulfils most of what people are asking for above and on the previous PR:

  • Defining your own delimiter lets you more easily wrap any desired string content.
  • Very easy to READ exact content of string. No bytes between the delimiters that are not in the string other than leading and trailing newline
  • Very easy to paste an exact byte-for-byte document without having to reformat/rewrite in any way.

All this is to say maybe giving HEREDOC-esq syntax a second look would meet the actual needs of the community better and more cleanly than anything else being discussed here.

Yes, HEREDOC syntaxes can get complicated if you let them, but you don't have to let it get complicated.

If you just avoid the indentation options and other pitfalls other languages heredocs add complication with, it's fine and easy to read.

@MOZGIII
Copy link

@MOZGIII MOZGIII commented Feb 13, 2020

Btw, SQL use case, in particular, can be solved not via string, but via something like "include file" macro. It's kind of better in some sense, cause it's possible to compile it in such a way that the embedded file is in it's own section, and probably other smart things can be done to improve it. Of course, seeling the actual SQL along the code worth a lot more than some binary layout optimizations, so I'd vote for using the fixing the string. Although, maybe a modern editor can display the embedded file inline - then it becomes a to-go solution again...

@alanfo
Copy link

@alanfo alanfo commented Feb 13, 2020

Vis-à-vis HEREDOC style syntax, it's worth remembering that, even with the present raw string syntax, one can still just paste text straight in without further ado.

This is because in many of the contexts where raw strings are typically used a leading/trailing new line is unimportant though they can easily be trimmed off as the following example shows:

package main

import (
    "fmt"
    "strings"
)

func main() {
    s := `
Line 1
Line 2
Line 3
`   
    fmt.Printf("%d\n", s[0]) // 10 ('\n')

    t := strings.TrimSpace(`
Line 1
Line 2
Line 3
`)

    fmt.Printf("%d\n", t[0]) // 76 ('L')
}

Under my idea of just introducing """ as an alternative raw string delimiter to enable embedded back-ticks to be used, the position would be exactly the same.

Indentation is a problem for any approach though this is best dealt with IMO by adding a suitable function to the strings package rather than trying to do something within the language itself.

@alanfo
Copy link

@alanfo alanfo commented Feb 13, 2020

@MOZGIII

In my practice, it happened at least once, in a non-source-code context.

Can you remember the context(s) in which back-ticks and triple double quotes occurred in the same raw string?

I've been trying to gauge what we'd be missing by having a simple solution to this issue and, so far, the answer has been - not much.

@MOZGIII
Copy link

@MOZGIII MOZGIII commented Feb 13, 2020

It was very narrow case where that sequence was explicitly required to be displayed, unlikely someone else stumbles upon this. That said - I've had way more cases where backticks and/or triple quotes been isufficient for embedding code in them. They're not a solution for a generic case, that's a fact, but I've already said everything I had to say about this above.

@alanfo
Copy link

@alanfo alanfo commented Feb 13, 2020

So it appears that the only real difficulty is embedding source code in other languages and potentially in Go itself.

I wonder how common this is in the wild? In most cases, I suspect that the amount of source code would be such that you'd want to load it in from a file.

@MOZGIII
Copy link

@MOZGIII MOZGIII commented Feb 13, 2020

Almost every time I embed multi-line strings is to embed something that's code. 95%. I'd say it's a major use case. Very rarely I embed verbatim text paragraphs without typography (i.e. without html/markdown). And I have no idea what else can even be possibly considered not code.

@deanveloper
Copy link
Author

@deanveloper deanveloper commented Feb 13, 2020

I think an issue with having two separate delimiters to represent the same thing is that people may get curious as to what the difference between backticks and triple-quotes/heredocs/etc, especially if they have dealt with languages like Bash where every kind of quote means a different thing. Finding answers online can also often be difficult, as people may often have trouble putting their question into words, and there are always many answers, even if the answer in the end is simply "they mean the same thing".

So the huge advantage to delim`string`delim and ``string`` syntaxes is that they are similar to the existing raw string syntax, meaning that intuitively, they are no different from the single-backtick raw strings except for their delimiters.

@alanfo
Copy link

@alanfo alanfo commented Feb 13, 2020

Well, by source code, I meant code for general purpose programming languages as opposed to: SQL, XML, json, mark-up etc.

But, if embedding code is as common as @MOZGIII says it is, then the simple solution I was proposing is not really good enough.

FWIW, of the '100%' solutions I prefer the:

``"this is a `raw` "string" literal"``

idea which I think @ianlancetaylor came up with.

@alanfo
Copy link

@alanfo alanfo commented Feb 13, 2020

@deanveloper I don't really understand your point as ISTM that, no matter how you look at it, you're going to end up with two different types of delimiter for raw strings to solve the back-tick problem.

If one accepts that the problem is worth solving at language level, there are basically two questions to answer:

  1. Is having a constant alternative delimiter which is the same at both ends good enough to deal with nearly all of the cases which might occur in practice bearing in mind this approach can never deal with 100% of cases now matter how rare the chosen delimiter may be.

  2. If a constant delimiter isn't good enough, how variable does the second delimiter type need to be. The more variable it is the more difficult it is to learn, read, lex and describe in the spec.

As you will have gathered, I was hoping that 1. might be good enough but I'm now not so sure.

@tv42
Copy link

@tv42 tv42 commented Apr 17, 2020

@deanveloper Please stop repeating that something you disagree with is "poorly designed", you're trying to dress an opinion in words that imply more than it being just an opinion. Simple is great and I love the design of simple things, in computers and in real life objects.

Raw strings are designed to store large amounts of text

[citation needed]

I would argue that "large amounts of text" don't belong inside a Go source file. Personally, I have large SQL queries in separate files and use asset embedding techniques.

@DeedleFake Disallowing empty raw strings is not realistic, they're out in the wild and justified well enough. For example:

// foo.Printf adds a newline (/otherwise sends a "complete
// message"), like log.Printf
foo.Printf(`And then %s said, "Hello".`, name)
foo.Printf(``)
func bar(start, stop string) ... { ... }

// 2nd arg is a regexp that matches the empty string, raw quotes for consistency
bar(`this\s+is\s+really\s+a\+regexp`, ``)
@verdverm
Copy link

@verdverm verdverm commented Jun 14, 2020

"RAW" strings in Cuelang work by using N # around triple-double quotes: https://cuelang.org/docs/tutorials/tour/types/stringraw/

I'm having the issue that I would like to embed markdown strings in my Go program as strings that I could process and server (like a web version of docs / help flags).

Perhaps there is a better asset embedding technique for Go programs that is recommended? Something that compresses it, embeds it, and then decompresses into mem when running?

@ncw
Copy link
Contributor

@ncw ncw commented Jun 16, 2020

I find myself embedding markdown frequently in Go code and not being able to put a backquote into a backquoted string is a minor annoyance.

I think this proposal is probably too complicated though.

I think we should just copy what Python does and have triple quoted strings, both """ and ```. Having a triple quoted " means that it can include new lines and " and also escape characters for when you really really need to insert """ into it!

This easy to lex, backwards compatible, familiar to Python users, simple to explain and covers nearly all use cases without getting overcomplicated.

@niklas-ourmachinery
Copy link

@niklas-ourmachinery niklas-ourmachinery commented Jul 28, 2020

I didn't see Lua's approach mentioned here, which I personally like. Lua lets you delimit a raw string with [[ and ]]. In the case where ]] is used in the string, you can add an arbitrary number of equal signs = between the delimiters: [=[string with ]] in it ]=]. You would only need more than one equal sign if the string contained ]=].

Things I like about this solution:

  • I find it less ugly than prefix strings.
  • Any raw string can be quoted, it is just a matter of adding enough equal signs between the delimiters.
  • Start and end delimiters are different, so you don't run into the problem with an even number of backticks.
  • We still only use a small set of characters as delimiters, it is not totally arbitrary delimiters, which makes the code easier to read.
@deanveloper
Copy link
Author

@deanveloper deanveloper commented Aug 7, 2020

I did not think about the benefits as mentioned in #40393. To summarize that issue, named backtick literals would also allow for syntax highlighters to highlight the raw strings differently, which is especially useful for SQL/HTML strings.

@smasher164
Copy link
Member

@smasher164 smasher164 commented Sep 6, 2020

I have come to really appreciate Rust's raw string literals. I don't think we need to go as far as an arbitrary identifier as a delimiter. Such a change would warrant an addition to the type system to assign a name to forms of literals, which gets complicated. The design should be driven by the smallest addition to the current backtick literals that does not make lexing immensely more complicated.

I think we can further simplify Rust's string literal syntax by excluding the r prefix before the delimiter. This requires choosing an appropriate character for the variable delimiter that is obvious during lexing. Using a character not currently used anywhere in the grammar would ensure this, like '~', '@', '#', '$', or '?'. OTOH, there are some unary operators in the spec that could be reused, like '+', '-', '!', '^', '*', or '&'. Repetition of these is currently permitted, so once a backtick is encountered, we use the repeated unary_op as the delimiter for the string.

I quite like the second option, and I'm in favor of using the unary '^' operator as the delimiter:

const s = ^^`
func main() {
	sql := ^`SELECT `foo` FROM `bar` WHERE `baz` = "qux"`^
	fmt.Println(sql)
}
`^^
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.