Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: Go 2: improvements to raw strings #32590

Open
deanveloper opened this issue Jun 13, 2019 · 69 comments
Open

proposal: Go 2: improvements to raw strings #32590

deanveloper opened this issue Jun 13, 2019 · 69 comments

Comments

@deanveloper
Copy link

@deanveloper deanveloper commented Jun 13, 2019

Background

This proposal was branched off of #32190, which was a proposed HEREDOC syntax for Go. It was concluded that HEREDOC was not the correct syntax for Go to use, however the proposal did point out a large problem that Go currently has:

Problem

There is only one option to use raw strings, which is the backtick. The nature of how raw strings works means that raw strings themselves cannot contain backticks, meaning that the current workaround for including a backtick in a raw string is:

var str := `My backtick is `+"`"+` hard to use`

Raw strings are often used for storing large strings, such as strings containing other languages, or Go code itself. In many languages, the backtick has significant meaning. For instance:

  1. SQL uses backticks to signify a string that represents an identifier, such as a database name or table name. While they are not required for all identifiers, they are required if the identifier contains invalid characters (spaces, commas, etc) or if the identifier matches names with a keyword. It also seems to be good practice in general to surround database and table names in backticks.
  • Example: SELECT * FROM `database`.`table`
  1. Kotlin uses backticks in a similar fashion.
  • Example: fun `a method with spaces`() { ... }
  1. JavaScript uses backticks to indicate format strings, which allow people to embed expressions inside of their strings.
  • Example: let str = `HELLO ${name.toLocaleUpperCase()}`

Of course there are far more examples of languages where the backtick is a significant character in the language. This makes embedding these languages in Go very hard.

Proposed Solution

If there were a fixed number of ways to declare raw strings, the problem would, no matter what, arise that you would be unable to put Go code inside of Go code without some kind of need to transform the code. This means that there needs to be a variable way to create raw strings.

This proposal highlights one brought up here. It essentially improves on the current way to declare raw strings, allowing the following syntax:

var stmt = SQL`
SELECT `foo` FROM `bar` WHERE `baz` = "qux"
`SQL

var old = `
this, of course, still works
`

var new = 고`this
also works
    you can also use 고 AND `backticks` (separately) in the string!
`고

Essentially, raw strings can be prefixed with a delimiter, and the string is then terminated with a backtick followed by the same delimeter.

Strings which are densely populated with words and backticks may make it hard to pick a word to use as the delimiter for the raw string, as the word may appear inside the string, which would end the string early and cause a syntax error. Allowing any identifier to be used as a delimiter would allow non-ascii characters to be used as well, meaning that in special cases, when it's really needed, one can use a non-ascii character as their delimiter.

Concerns

@jimmyfrasche #32190 (comment)

Implementation-wise, the problem with user-specified delimiters is that they have to be handled during lexing, which adds complexity to a simple, though still somewhat involved, stage and would need a lot of explanation in the language spec.

I don't like the idea of complicating the language. I do not work with the internals of the language, so I am unsure of the magnitude of complication to the lexer that this change would bring. If it is too much, I don't think that it would at all be worth it, and maybe one of the alternatives below would be a better fit.

@ianlancetaylor #32190 (comment)

My only concern with [syntax] is that it doesn't lead with the fact that it is a string. C++ (R"delim( string )delim")) and Rust (r#" string "#) and Swift (#" string "#) are more clear as to when a string is starting.

I share this sentiment. My response to this here was that establishing a convention to use short, noticable identifiers (ie RAW, JS, SQL, etc) help with noticing where the string starts and ends. This could (possibly) be enforced by golint, but I'm not sure if that is a good idea or not.

Other Alternatives brought up

In #32190, there were several other alternatives that tried to achieve the same goal:

Variable numbers of backticks

Essentially, you could start the raw string with a certain number of backticks, and it would have to end with the same number of backticks.

    `````
    A raw string which can contain up to 4 ```` backticks in a row inside of it
    `````

This solution still had problems though. Strings cannot start with an even number of backticks, because any even number of backticks could also be interpreted as an empty string, introducing ambiguities. It also causes developers a bit of fuss when trying to get it to work inside of markdown, as markdown uses multiple backticks in order to signify a block of code.

Also, the strings could not start or end with backticks, which would be an unfortunate consequence.

Variable number of backticks + no empty raw strings

This one is a breaking change, however I think it is my favorite solution out of all of the alternatives. It's the exact same as the previous one, but Go also introduces a breaking change to disallow empty raw strings. There is no need for raw strings to be used to represent an empty string, since the normal "" can do that, and is much more preferable. The only code this would break is people who have used a raw string to define an empty string by doing something like x := `` or funcCall(``, ...). It may be good to do some research on if empty raw strings are ever used in real code.

This solution still has the issue of being annoying to use with markdown's code fences. The argument was used that we shouldn't make language decisions based on other languages, however I personally do not like this argument. Sharing code is part of what a programmer does, and Markdown is a very widely used markup language that uses multiple backticks in a row to define a code fence. This feature may make it a bit difficult to share Go code over anything that uses Markdown (slack, github, discord, and other services).

Despite making it difficult to share code via markdown-enabled chats, it is still easy to share code via something like gist.github.com or play.golang.org. If my original proposal proves to not work very well (doesn't feel Go-like, too difficult to implement, etc) I would love for this solution to be accepted in place.

Variable number of backticks + a quote

This proposal is actually pretty nice. It's similar to the previous proposal. Essentially, the starting is N backticks (N >= 2) followed by a quotation mark, and the ending delimiter is a quotation mark followed by the same number of backticks. Example:

s := ``"this is a `raw` "string" literal"``
fmt.Print(s)

// prints:
// this is a `raw` "string" literal

This syntax is actually very nice in my opinion. It fixes the "odd-number-only" ambiguity from the previous example, as well as fixing the Markdown issue (as code fences must occur on their own line). It also fixes the "strings starting/ending with backticks" issue.

The only issue with this syntax is that it doesn't seem to work well with existing raw strings. I don't personally have data about how often this occurs, but I'd imagine that there are several times where raw strings are used to describe strings with quotes in them, making code like x := `"this is a string"` common. Newcomers to Go may see this and think that the `" is the delimiter to the raw string, when in reality the ` is the delimiter and the " is part of the string.

However that critique may be a bit nitpicky. I do like this syntax a lot.

Choosing a symbol pair that nobody uses

This alternative stated that Go should add another symbol to use to declare raw strings in Go. For instance, to start the string and to end the string. Go code is defined to be UTF8 so file formatting issues should not happen. Another proposed idea was (U+2261 IDENTICAL TO).

This solution also has problems. What if our string has both backticks AND strange symbols (for instance if you were defining a list of mathematical symbols)? Or, what if you were trying to embed Go syntax inside of your strings? Also, the symbol is hard to type and not easy to find, so it may not be a good fit as a string delimiter.

Other languages

  1. C++ R"delim(string)delim"
    • In my opinion, I personally hate the asymmetry of prefix-strings, they look sloppy to me and seem too much like they were trying to hack in features, so I don't really like this solution.
  2. Rust r#"string"#
    • Same issue that I had with C++: the asymmetry and "hackiness" of prefix-strings ruins it for me. Also, a fixed number of ways to define a string means that if one wants to put a Go raw string inside of a string (ie pattern matching for code generation), they will run into issues.
  3. Swift #"string"#
    • Again, a fixed number of ways to define a string means it's hard to pattern-match Go raw strings for code generation.

It's important that we have some kind of variable delimiter, as that way if the string we are embedding somehow contains it, it is easy to change the string's delimiter in order to avoid the issue.

The delimiter doesn't have to be an identifier like it is in this main proposal, it could also be varying the number of backticks like the one a few paragraphs up.

Conclusion

Raw strings in Go are often used to be able to copy-paste text to be used as strings, or to embed code from other languages (such as JS, SQL, or even Go) into Go. However, if that text contains backticks, we need some way to make sure that those backticks do not terminate the string early.

I believe that the way to do this is allowing an identifier to precede the string, and to make sure that the terminating backtick must be followed by the same identifier in order to terminate the string.

var markdown = MD`
### Thank you for reading :)
`MD
@gopherbot gopherbot added this to the Proposal milestone Jun 13, 2019
@gopherbot gopherbot added the Proposal label Jun 13, 2019
@deanveloper

This comment has been minimized.

Copy link
Author

@deanveloper deanveloper commented Jun 13, 2019

Just found this as well: #24475

I did not see this one while making this proposal, my bad. But it does contain another interesting suggestion by @bcmills #24475 (comment)

Specifically, we could treat as a raw string any sequence of characters which:

  • begins with a QUOTATION MARK character in Unicode catogory Pi, and
  • ends with the corresponding QUOTATION MARK character in Unicode category Pf.
@davecheney

This comment has been minimized.

Copy link
Contributor

@davecheney davecheney commented Jun 13, 2019

SQL uses backticks to signify a string that represents an identifier, such as a database name or table name.

I do not believe this is correct. Certainly for databases like MySQL and Postgress their quoting character is ', ASCII 0x27.

Ref: https://dev.mysql.com/doc/refman/8.0/en/string-literals.html Section 9.1.1
Ref: https://www.postgresql.org/docs/11/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS Section 4.1.2.1

@cespare

This comment has been minimized.

Copy link
Contributor

@cespare cespare commented Jun 13, 2019

@davecheney these databases use single quotes for strings, yes. (Some allow " as well.)

Backticks are used by MySQL (and sqlite? any others?) if you need to quote identifiers: https://dev.mysql.com/doc/refman/8.0/en/identifiers.html (search for "backtick").

(Postgres uses " for this purpose.)

@cespare

This comment has been minimized.

Copy link
Contributor

@cespare cespare commented Jun 13, 2019

I like @deanveloper's main proposal, but on reflection I like @ianlancetaylor's idea (``" a `raw` "string" "``) even more. It is less flexible than using an arbitrary identifier, but that extra power seems largely pointless and introduces another place where people will have to choose/argue about a name.

@davecheney

This comment has been minimized.

Copy link
Contributor

@davecheney davecheney commented Jun 13, 2019

@cespare Thank you for the correction.

I continue to assert that back ticks for SQL string construction is not a valid argument for a language change proposal -- users should not be constructing SQL query strings by hand. The fact that

a. Alternative syntax like ' (which I believe is part of the SQL standard, but I couldn't find a linkable reference) is supported
b. You only need to use back tick in cases where you have to quote reserved words, and the alternative exists now +""+`. Its not pretty, but that's because people shouldn't be constructing SQL strings by hand.

Gives weight to this position.

@deanveloper

This comment has been minimized.

Copy link
Author

@deanveloper deanveloper commented Jun 13, 2019

@davecheney I gave other examples, such as JavaScript and Go, which are both languages that I can see being put into raw strings. JavaScript, of course because Go is used a lot for web backends, so, depending on the circumstance, it could make more sense to embed a small script into a raw string rather than create an entire separate file for it. Go may make sense for code generation purposes if you wish to detect a raw string in a file.

Also, there is the possibility that someone may just need to encode a string that has many backticks in it.

I still personally believe that SQL is a valid use-case for this. SQL Injection attacks are caused by people using string concatenation in SQL queries, not necessarily because they are crafting queries by hand. The database/sql library actually takes care of sql injection with the ? syntax, which allows people to pass in user-generated arguments with automatic sanitation of inputs. It's a pretty well-known feature as far as I'm aware. Asking all users to use third-party libraries in order to query tables is unreasonable IMO, especially when Go already comes with standard library support for database/sql.

a. Alternative syntax like ' (which I believe is part of the SQL standard, but I couldn't find a linkable reference) is supported

Double quotes are technically supported by MySQL, but only if ANSI_QUOTES mode is enabled (as shown by the page linked by @cespare). Enabling this mode also requires that string literals no longer use double quotes.

Also, while backticks are not technically required in SQL languages, they are still common practice. If a DBA sends me a complex query to use, I don't want to go through the whole thing and remove each backtick, or need to put in a `+"`"+` instead.

@MOZGIII

This comment has been minimized.

Copy link

@MOZGIII MOZGIII commented Jun 13, 2019

@davecheney, backticks are not in the SQL standard, they're database specific. For example MSSQL doesn't use backticks, but MySQL and Postgres do. With those DBs there's no alternative to using backticks sometimes.
Often you just need to encode an SQL string literal without any dynamic construction (i.e. no variable substitution or conditional logic). This is not considered "constructing SQL by hand" and is, on the contrary, very recommended, while usage of query builders and ORMs is somewhat discouraged in Go.

@davecheney

This comment has been minimized.

Copy link
Contributor

@davecheney davecheney commented Jun 13, 2019

while usage of query builders and ORMs is somewhat discouraged in Go.

Could you please cite a source for this position. Thank you

@davecheney

This comment has been minimized.

Copy link
Contributor

@davecheney davecheney commented Jun 13, 2019

To clarify my position. If you’re using sql you should be using prepared statements. To the best of my knowledge when that is done the dB drivers take care of quoting for identifies and values. This is why I assert sql queries are not a good premise for this proposal.

@deanveloper

This comment has been minimized.

Copy link
Author

@deanveloper deanveloper commented Jun 13, 2019

Could you please cite a source for this position. Thank you

While this is not a direct source (I don't have one, but I also am not the one who claimed this fact so I think I get a bit of leeway):

The lack of support for generics is a huge bummer for SQL Builders, because they cannot have type safety, which is really the only advantage they had over ? syntax. They also typically heavily rely on chaining syntax which Go also discourages with it's semicolon placement rules. This was also referenced in the design doc for error handling here (scroll to the ? operator question). While not stating that chaining is discouraged, definitely stating that chaining is uncommon. Again, this is because of semicolon placement rules, which cause chaining (aka builder syntax) to become a bit ugly:

var builder myBuilder
x := builder.
    AddThis().
    AddThat().
    SetFoo(bar).
    Baz().
    Build()

This is not a problem for simple queries, however as queries get more complex, it gets exponentially worse.

To clarify my position. If you’re using sql you should be using prepared statements. To the best of my knowledge when that is done the dB drivers take care of quoting for identifies and values. This is why I assert sql queries are not a good premise for this proposal.

Literals are common in SQL queries, in which case I need to worry about quoting. I don't want to replace every single one of my literals with a ?. For instance, a simple query for selecting every user who used a certain coupon:

SELECT `user` FROM `users` JOIN `transactions` USING (`uid`) WHERE coupon_applied = "someCoupon"

There are definitely times where I want string literals embedded in my queries, and again as queries get more complex I definitely don't want to be moving string literals into ?.

@MOZGIII

This comment has been minimized.

Copy link

@MOZGIII MOZGIII commented Jun 13, 2019

Well, the way Go standard library (database/sql) is implemented is clearly in favor of writing the SQL literals. The reason for that is if you use some form of SQL builder or ORM you often don't pay enough attention to what's going on under the hood. This, in turn, leads to nasty performance and code logic issues.

Prepared or not, you still need to communicate your statement (aka query) to the database server, and this usually requires writing your own SQL literal string. Sure it'd be a big mistake to use string concatenation instead of binding the query params. Binding works for both usual and prepared statements. Btw, prepared statements have their own issues - for example they're very hard to manage with transaction based connection pooling (i.e. via pgbouncer) - so I tend to not use them that much. Performance wise, in many typical cases prepared statement slow down the overall system performance, rather than speed it up - so use them with caution and measure the effect.

@davecheney

This comment has been minimized.

Copy link
Contributor

@davecheney davecheney commented Jun 13, 2019

@MOZGIII i disagree with most of your advice, yes prepared statements have a cost, but a very clear benefit, and you don't need to quote the ?'s which is my point.

There must be a better use case for this proposal than SQL strings.

@deanveloper

This comment has been minimized.

Copy link
Author

@deanveloper deanveloper commented Jun 13, 2019

@davecheney I've mentioned both in the original proposal and my first reply to you other possible use cases, but you never seem to address them. Even if they're relatively minor, there's just no good way to encode a multiline string that contains backticks. Is that really too much to ask, especially out of a general-purpose programming language?

@davecheney

This comment has been minimized.

Copy link
Contributor

@davecheney davecheney commented Jun 13, 2019

@deanveloper if we take SQL quoting out of the proposal, then the use cases you're suggesting are writing snippets of other languages in Go string literals? How common is this in the wild?

@MOZGIII

This comment has been minimized.

Copy link

@MOZGIII MOZGIII commented Jun 13, 2019

@davecheney When variable binding is used, the actual variables and the query are typically passed as separate units to the database engine. I can assure that it is the case for postgres, but less advanced databases actually may do something different. But nonetheless, in postgres, the data packet to the database over the wire can be represented as the following tuple: (string, Value[]). If you use the SQL literal SELECT ?::text it will be sent over the wire as is (pretty much). Along with it, the bound variable will be sent, for example "hello world". Moreover, the bound value will be encoded with the proper wire format for strings. On the server side, the database' will first run the received statement through the parser, then match the ?s to the values and then execute the actual query. Nowhere in this sequence the value is quoted or put in place of the ? in the actual query. This is what we call variable binding. One of the additional bonuses of this approach is that database parser layer typically caches the parsing for statements (the statements tend to repeat a lot with this approach) - and it boosts the performance quite significantly.

Now, prepared statements. They can also use variable binding, but they don't have to, the same way as the regular, non-prepared statements. The problem with prepared statements is it's quite difficult to manage them on a per-session level, and nobody does that. The alternative is preparing the statement for every operation independently. This is slow, because it often causes multiple round trips. This may become a critical bottleneck, as it easily doubles the execution time in a well optimized system. As I said, variable binding is not only possible with the prepared statements. Under the hood it is implemented even for a regular SQL queries: https://github.com/jackc/pgx/blob/762e68533f0090ecb6bb1166d51966b326597ec7/query.go#L410-L453
This code completes in only one round trip, as long as all value types (OIDs) are known upfront.

Anyhow, let's get back to literals - I don't want this to be an SQL discussion. I would say that the lack of support to just copy and paste arbitrary string (SQL with quotes in particular) is a design flaw of the existing implementation. Especially while iterating on the whatever thing you need to represent as a literal - it's much easier to set the boundaries once rather than escaping the backticks on every iteration.

@MOZGIII

This comment has been minimized.

Copy link

@MOZGIII MOZGIII commented Jun 13, 2019

@davecheney Embedding markdown is another example. I feel like pretty much every other language that has backticks as part of the syntax may cause issues if we attempt using it in the current raw string literals implementation. Those are bash, ruby, perl, TeX and many more. You never know all the ways the feature that's not in the language could've been used. Frankly, even for features that are in the language it's very hard to tell all possible uses.

I think we should not accept "SQL is not a good example" as an argument. The rationale is a) the lack of a better example doesn't prove there's no problem, and b) SQL is not just a practical example, it is an actual pain point that I've encountered while solving my day-to-day tasks with Go.
Surely, more examples can shape the way we address the problem, but, unless we address the SQL pain point separately (which I don't see proposed here), I don't see why would we want to ignore it.

@deanveloper

This comment has been minimized.

Copy link
Author

@deanveloper deanveloper commented Jun 13, 2019

@davecheney I'm going to come at this from a different angle, actually. Regardless of if you want to use Go for code generation, which I personally do quite a bit, raw strings the way they are now are just poorly designed.

Raw strings are designed to store large amounts of text, or text which contains special characters (" and \), or even both, into a variable. However, if the string contains a different special character, then the entire purpose behind raw strings falls apart. At least in other languages they have safeguards such as using two different characters in a row as a delimiter to a string, making it quite a bit harder for the string to be terminated early. However, even this solution still has the problem with code generators being a bit messier.

@alanfo

This comment has been minimized.

Copy link

@alanfo alanfo commented Jun 13, 2019

The more argument (and lack of agreement) I see on this matter, the more convinced I become that it's not worth doing anything here and we should just live with what we have.

The present syntax for raw strings has the desirable properties (for a simple language such as Go) of being very easy to type, parse, read, explain, understand and remember. The only problem it has is not being able to include the back-tick character and opinions differ on how acute this problem actually is in the wild.

All the proposed solutions (including those suggested by myself in #32190) either seriously compromise one or more of these properties and/or deal with the back-tick problem at the expense of creating a similar problem with some other (more rarely used) character.

It's also worth remembering that, apart from the workaround mentioned by @deanveloper in his opening post, there are (at least) two other ways of dealing with the back-tick problem at the present time:

  1. Changing the back-ticks in the raw string to some other unused character and then applying strings.Replace on the resulting string to change them back to back-ticks.

  2. Using fmt.Sprintf in the following manner:

    s := fmt.Sprintf(`This is a "raw" string including %cback-ticks%[1]c.`, '`')
    fmt.Println(s)
    // This is a "raw" string including `backticks`.

Admittedly none of these solutions is exactly "nice" but, on balance, I think they are better than introducing some new (and probably contentious) syntax to deal with the back-tick problem which I don't think everyone regards as being particularly serious in any case.

@MichaelTJones

This comment has been minimized.

Copy link
Contributor

@MichaelTJones MichaelTJones commented Jun 13, 2019

This has my support. I'm a voice for Unicode as an out of (ASCII) bounds signalling means, but admit that while it solves the easy case of quoting all normal situations, it cannot alone address recursive self quoting; for that we must have a case-specific delimiter, and identifier-backtick is a clean, simple, universal mechanism.

It is trivial to parse so I dispute any pushback on inconvenience of implementation. In addition to the likely uses...

`say, "plugh"`

xxx`say, "plugh"`xxx

 Δ`say, "plugh"`Δ

...it also allows an extreme case: generate a huge random integer (256 bits, say), encode that in identifier form (letter or underscore followed by (letter|underscore|digit)*), and use that to blind quote text without looking at it and knowing it will work. This may never come to pass, but I like that it could:

randomQuoteTag2f4d87ef38eb88c21a00104006339a15e9cbe43f2610c247314cef54ab5f5db8`unlikely to have collisions`randomQuoteTag2f4d87ef38eb88c21a00104006339a15e9cbe43f2610c247314cef54ab5f5db8

This is something you could rely on when, say, packaging files in strings, like a go generate tool that pastes a copy of a source file into that source file.

@deanveloper

This comment has been minimized.

Copy link
Author

@deanveloper deanveloper commented Jun 13, 2019

Admittedly none of these solutions is exactly "nice" but, on balance, I think they are better than introducing some new (and probably contentious) syntax to deal with the back-tick problem which I don't think everyone regards as being particularly serious in any case.

My personal issue with this solution is that you then must apply transformations to the string before the code can compile. This is especially undesirable if you do not have syntax highlighting, which is a crutch that Go should not depend on. It is also annoying to write a program to apply the string transformation for you, as you cannot put the original string into a Go program.

@kardianos

This comment has been minimized.

Copy link
Contributor

@kardianos kardianos commented Jun 13, 2019

@deanveloper I write a lot of SQL and SQL system and I write a lot of Go. I'm neutral on the core of this proposal. However the motivation for a change is important.

When building SQL strings, you may want to quote identifiers and quote text/time/(a few other cases) values. It is assumed that you will use value parameters where appropriate. Also of note, a prepared query and and a query that uses value parameters are not directly correlated in most system. Different database system support different ways to quote identifiers and values. SQL Server uses square brackets [] to quote identifiers and single quote marks for values only. PostgreSQL uses double quote marks for identifiers and single tick marks for values.

Let's take your SQL problem and address it first. For this problem, the solution you have will work, but is not one I would recommend. It is too ad-hoc for a system. Spend a half a day and we can make a better solution.

Spend a little bit more time in discovery to find other motivating pain points. The example you gave just isn't that motivating to me.


A better solution

package sqls

import (
    "github.com/golang-sql/sqlexp"
)

// Replacer sets up replacement classes for SQL strings.
type Replacer struct {
    Quoter sqlexp.Quoter // If nil, ANSI SQL is assumed.
    IdentiferReader rune // If zero value, ':' is used.
    VariableReader rune // If zero value, '@' is used.
}

// Replace reads a format string and replaces identifier names that start with the IdentifierReader
// followed by by at least one letter class rune and zero or more letter or number class runes.
// Similar for VariableReader names. Values may be either a map[string]interface{} or a typed
// struct. If a struct is used reflection will be used.
func (r Replacer) Replace(f string, values interface{}) (string, error) {
   //...
}

func TestReplacer(t *testing.T) {
    r := &Replacer{}
    out, err := r.Replace(`
select
    :MyColumns/t1,
    :OtherColumns/t2
from
    :MyFirstTable t1
    join :MySecondTable t2 on t1.ID = t2.:LinkColumn
where
    1=1
    and :Where1/t1 = @MyValue
`,
    map[string]interface{} {
        "MyColumns": []string{"ID", "Name"},
        "OtherColumns": []string{"BookName=Book"}, // Identifiers are inspected for "=" for alias.
        "MyFirstTable": "Author",
        "MySecondTable": "Book",
        "LinkColumn": "AuthorID",
        "Where1": "Location",
        "MyValue": "US",
    })

     fmt.Println(out)
     // OUTPUT
     /*
select
    t1."ID", t1."Name",
    t2."Book" as "BookName"
from
    "Author" t1
    join "Book" t2 on t1.ID = t2."AuthorID"
where
    1=1
    and t1."Location" = 'US'
*/
}

There are lots of ways to solve the problem you presented. This would be one of them. I would argue it would do a better job of what you are wanting then what raw strings (current or improved) can give you. You could implement this as a runtime step or as a pre-compile step. I've also used text/template + some custom template functions to construct SQL.

Constructing SQL is fine. Carefully including values in SQL text can be okay if done through a system (like above) to prevent silly mistakes. I don't care if you use parameters or not, I don't care if you prepare your SQL or not (modern SQL engines a prepare is useless).

@deanveloper

This comment has been minimized.

Copy link
Author

@deanveloper deanveloper commented Jun 13, 2019

Again, SQL is not the only target of this. Raw strings as they are now, are poorly designed as illustrated in #32590 (comment). That comment is a better illustration of the problem that we currently face than all of this arguing on SQL Queries and how they should be done.

@MOZGIII

This comment has been minimized.

Copy link

@MOZGIII MOZGIII commented Jun 13, 2019

@kardianos that, or there could've just been a string literal. It is a beautiful workaround for a problem that shouldn't exist in the first place. With this you can either do a compile-time codegen - in which case just having a .sql file wrapped with Go literals with proper escaping is sufficient, and the end result can be a statically allocates string from the binary image - or it can be applied at runtime - probably during the system initialization, cause actually constructing an SQL query like that for every operation would result in lost cycles (unless you actually generate different strings - which is possible, but comes close to what I'd avoid doing in the first place, as the dynamic values, either from the user or from the system itself, we always use the standard API to pass them to the driver via Args list).

The problem with the example above is that, while it solves one part of the problem - and that is allowing you to get a program that invokes the query you want - it doesn't address other parts - among those are the ability to author the same literal that the app will use. This is more important for literals talk than the ability to actually somehow make the string value that you want appear at runtime. Go already addresses this problem, it just does it poorly (we have the ` notation in the language).

As a counter example of why we actually need to have good literals for raw strings: any byte array (even non-unicode) we can represent in a Base64 encoding, embed into the regular " string (no need for even `), decode it in runtime and be happy with it. The downsides of this solution should be obvious.

@kardianos

This comment has been minimized.

Copy link
Contributor

@kardianos kardianos commented Jun 13, 2019

Spend a little bit more time in discovery to find other motivating pain points.

@kardianos

This comment has been minimized.

Copy link
Contributor

@kardianos kardianos commented Jun 13, 2019

@MOZGIII Any string literal by itself won't automatically escape value or identifiers, nor will it expand an array of columns nor will it escape and join together a list of strings or other values. I personally find these things important. You may not.

@MOZGIII

This comment has been minimized.

Copy link

@MOZGIII MOZGIII commented Jun 13, 2019

@kardianos so, what you mean is your code does something valuable that's not directly correlated to representing literals, and as a bonus it solves the issues with `. That's great, but what if you don't need all that other stuff? I personally sometimes just need to put whatever SQL I have agreed on with my peers in to the code - and with that workflow it's important that the query is actually the same that was used in prototyping so other people are familiar with it. Especially under the conditions where time is money and the SQL is typically 20+ lines. That said - those are all second order issues, and they key thing here is to improve the literals the language provides.

Are we still in the "why?", or are we in the "how? already? If we're in the "how?" then I don't see the point of looking into the workarounds to the SQL use case - let's use that example to see what literals we can offer to solve the pain point.

PS: just noticed markdown supports multiple backticks for verbatim substrings. This means we can play around with this idea easily!

Example: (working)

1) `a`
2) ``a``
3) ```a```
4) ````a````
5) `` ` ``
6) ``` ` ```
7) ```` ` ````

Result:

  1. a
  2. a
  3. a
  4. a
  5. `
  6. `
  7. `

Example: (not working)

1) ``
2) ```
3) `````
  1. ``
  2. 
    

I must say I like qwe`value`qwe-style the most currently. Something like qwe"value"qwe or ##`value`## works too, as well as ```"value"```. To choose one among all the possible options, I guess the best way is to try writing some code pretending the change is already there and compare the solutions.

@rogpeppe

This comment has been minimized.

Copy link
Contributor

@rogpeppe rogpeppe commented Jul 10, 2019

@rogpeppe One of the main goals of this proposal is to make it so that raw strings can contain backticks.

I guess it depends whether "raw strings" are a goal in themselves, or whether we're really after a better way to represent arbitrary multiline text that can contain arbitrary characters including backticks.

@alanfo

This comment has been minimized.

Copy link

@alanfo alanfo commented Jul 10, 2019

It seems to be a cool idea though, however Go already has 2 "flavors" of strings, I'm not sure if introducing a third is a good idea.

It seems to me that the current proposal is in effect proposing the introduction of a third kind of string though admittedly it would be less of a departure from the current raw strings than text blocks would be.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jul 10, 2019

@rogpeppe I like the basic idea of indented strings, but I note that they don't have to be in the language itself. People could write

    s := strings.Indented(`|
             |this is a long
             |string with an indent
             |character`)

I think these come up rarely enough that adding the call to strings.Indented (or preferably a better name) doesn't seem too onerous to me.

This differs from the concept of a raw string literal that contains a backquote, which I think does have to be in the language itself.

@alanfo

This comment has been minimized.

Copy link

@alanfo alanfo commented Jul 10, 2019

@ianlancetaylor This is a similar idea to Kotlin's String.TrimMargin method which works well in that language and may be a better name for it.

They also have a String.TrimIndent method which automatically detects the minimal indent without the need for a special character.

@DeedleFake

This comment has been minimized.

Copy link

@DeedleFake DeedleFake commented Jul 10, 2019

@ianlancetaylor Though it's relatively minor most of the time, the problem with making it a function call is that it can't be a const.

@rogpeppe

This comment has been minimized.

Copy link
Contributor

@rogpeppe rogpeppe commented Jul 10, 2019

@ianlancetaylor Yes, I've used that technique in the past, but it's not entirely satisfactory.

As @DeedleFake points out, you can't make const strings that way.

Secondly, unless special knowledge of the strings package were baked into gofmt, it wouldn't be able to indent such strings appropriately to the surrounding code.

Thirdly, that API isn't great because such a string can be malformed, in which case you'd need either an error return, a runtime panic, or some string that ends up sensitive to indentation, of which no option is particular palatable.

Most importantly though, that technique doesn't address the fundamental problem this issue is trying to solve, which is that you still can't put a backquote in there, something that is addressed if this kind of string representation is in the language itself.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jul 10, 2019

Understood about the backquote issue; I was thinking of the function as a way to address your issue while the backquote issue would be addressed separately. I think that @donatj 's comment above is compelling: it's desirable to be able to simply paste a raw string into a Go program and just add appropriate quoting at the beginning and end. So I don't think your suggestion, though useful in various cases, addresses the real problem of this issue.

@MichaelTJones

This comment has been minimized.

Copy link
Contributor

@MichaelTJones MichaelTJones commented Jul 10, 2019

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jul 10, 2019

@MichaelTJones I don't hate it. But when reading the code I'd somewhat prefer to see immediately that I am looking at a string, rather than seeing iant, which looks like an identifier, and then seeing ", at which point I see that it is not an identifier, but is actually a string.

I think that most other languages have raw string literals that preserve that property: it's clearly a string from the start. (C++: R"delim( string )delim"; Rust: r#" string "#; Swift #" string "#).

@deanveloper

This comment has been minimized.

Copy link
Author

@deanveloper deanveloper commented Jul 10, 2019

@ianlancetaylor I can agree a bit with that, I think that @DeedleFake's suggestion may be a better syntax.

Empty raw strings are pretty useless and rarely used [citation needed], it might be a good idea to break a small amount of compatibility and disallow empty raw strings, and allow a variable number of backticks instead.

@MichaelTJones

This comment has been minimized.

Copy link
Contributor

@MichaelTJones MichaelTJones commented Jul 10, 2019

@agnivade

This comment has been minimized.

Copy link
Contributor

@agnivade agnivade commented Jul 11, 2019

I am sympathetic to @donatj's comment about pasting a raw string as I have encountered something very similar. In my case, it was editing an existing string rather than just copying it. I will just copy over my comment from #32190 (comment) again since this proposal is for general raw string improvement.

Comment copied below---

I had an html template as a raw string. Now in that template, I have <script> tags and javascript inside those. I was going to use a console.log() with the ${} notation for variables inside a <script> tag, and then immediately realized that I am stuck.

What would have been a nice

console.log(`time taken: ${performance.now()-start}`)

became

console.log(` + "`time taken: ${performance.now()-start}`);" +
`
@qiangli

This comment has been minimized.

Copy link

@qiangli qiangli commented Aug 23, 2019

I might be a little off track. How about something like this (https://github.com/jteeuwen/go-bindata) built into the language? It will solve all the probems here and what's more, a powerful feature for including static asset content into the code for many other uses.
Here is one approach:
The file names (any file not ending in ".go") in the same directory can be referenced as variables of []byte. the variable name is the file name without the extension (filepath.Ext()) it follows the same export convention also.
e.g.:

mypkg/mysql.sql
mypkg/Javascript.js
....
mypkg/xxx.go:


pakcage mypkg

const sqlQuery = string(mysql)
var script = string(Javascript)

"Javascript" is exported and can be imported and referenced as "mypkg.Javascript" in other packages.
build flag is required to turn on this feature where one can specify what file extesions are included/excluded.

A another less dramatic change is to introduce a standard package such as "text/bindata" and standard functions such as func Raw(name string) []byte.

@deanveloper

This comment has been minimized.

Copy link
Author

@deanveloper deanveloper commented Aug 23, 2019

That solution doesn't quite work since file names can contain many characters that don't fit into Go identifiers, such as spaces, quotation marks, etc. It would also cause confusion to many as the variable is not declared anywhere.

It is an interesting idea though, since it would result in a compile-time error if anything happened to the file.

(Also, just pointing out that your "text/bindata.Raw" looks just like "io/ioutil.ReadFile" 😉 )

Either way, I think that solution solution misses the point - raw strings in Go accomplish their goal poorly. They are made to capture exactly what is in between two delimiters, no different. Newlines, backslashes, quotation marks, ALL are contained in the string. Except backticks, which are a fixed delimiter.

This proposal aims to add some kind of variable delimiter (hopefully either a prefix/postfix identifier, or variable number of backticks) for raw strings in Go, which would assure that there is not a single possible string that isn't able to simply be pasted into a raw string without transformations.

@cespare

This comment has been minimized.

Copy link
Contributor

@cespare cespare commented Sep 26, 2019

Here is another real life situation that I don't believe has been mentioned yet.

The language defined by text/template uses " and ` for strings, just like Go, with the same semantics.

Inside a template, I want to write a string literal that includes many " characters. To avoid excessive escaping, I wrote the literal using a raw string with backticks. However, the template text is itself inside Go code, so this doesn't work. I ended up doing something like this:

var tmplText = `{{with $json := ` + "`" + `{"hello": "world"}` + "`" + `}}
{{$json}}
{{len $json}}
{{end}}
`

whereas with one of these proposals, I could just directly write the text I want to use. For example, using my favored syntax (@ianlancetaylor's):

var tmplText = ``"{{with $json := `{"hello": "world"}`}}
{{$json}}
{{len $json}}
{{end}}
"``
@integrii

This comment has been minimized.

Copy link

@integrii integrii commented Feb 9, 2020

Let's say I am writing some code that creates a database table. I created the table in MySQL workbench and want to put the exact SQL that workbench creates into my code. That code might look something like this:

CREATE TABLE IF NOT EXISTS `mydb`.`user` (
  `email` VARCHAR(255) NOT NULL,
  `hash` VARCHAR(64) CHARACTER SET 'DEFAULT' NOT NULL,
  `create_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `admin` TINYINT NOT NULL DEFAULT 0,
  `DisplayName` VARCHAR(255) NULL,
  PRIMARY KEY (`email`),
  UNIQUE INDEX `email_UNIQUE` (`email` ASC) VISIBLE)

Without some kind of unique EOF delimiter like the original proposal suggests, how would you do this?

  • Search/replace all backticks to quoted backticks with concatenation sourrounding them
  • Try to switch all backticks to single quotes or something and hope it still works
  • Load a file with the contents
  • convert it to base64 and decode it into a variable
  • ??? this is getting silly
@alanfo

This comment has been minimized.

Copy link

@alanfo alanfo commented Feb 9, 2020

Although I've been skeptical in previous posts about the need to do anything here, after staring at the latest slab of back-tick infested SQL I've concluded that we should try and come up with a simple solution as the current workarounds are just too painful.

Reading through the whole thread from the start, I think we may have lost sight of the original problem - allowing raw strings to contain back-ticks - and gotten bogged down with trying to find a solution which can deal with absolutely everything.

Suppose instead we introduce a new optional delimiter for raw strings namely triple double-quotes : """a `raw string` containing back-ticks.""". Apart from being able to include any number of back-ticks, the advantages of this would be:

  1. The delimiter is the same at both ends and therefore easy to parse.

  2. Its backwards compatible and people would probably prefer to continue using back-ticks as delimiters in the majority of cases as they're shorter to write.

  3. It is no problem if the raw string begins or ends with a single or double double-quote character.

  4. A number of languages use a triple double-quote as a raw string delimiter so this should be quite familiar to many developers and not therefore take much learning. These languages now include Java which is using them for its 'text block' feature which I linked to earlier. Give the size of the latter's code base you'd think they must be fairly confident that it's a good combination to use from a rarity point of view.

  5. The use of double quotes rather than some more esoteric delimiter meets @ianlancetaylor's point about it 'looking like a string'.

  6. The effect on the language spec should be minimal - it just needs to say that a raw string either begins and ends with a back-tick or with a triple double quote. The former can't include a back-tick but can include triple double-quotes. The latter can't include a triple double-quote but can include a back-tick so the two are complimentary and otherwise behave exactly the same.

The only disadvantage I can think of is that you wouldn't be able to include BOTH a back-tick and a triple double-quote in the same raw string. But how many times does that come up in practice? With the possible exception of language source code, very rarely indeed I'd suggest.

@MOZGIII

This comment has been minimized.

Copy link

@MOZGIII MOZGIII commented Feb 9, 2020

The only disadvantage I can think of is that you wouldn't be able to include BOTH a back-tick and a triple double-quote in the same raw string. But how many times does that come up in practice?

In my practice, it happened at least once, in a non-source-code context. 😄

But whatever the solution is - it's better than nothing. 😄 At this point, I'd just take it. Id' say, getting a more generic would be more preferable to me, but it can be deferred till after we solve the major pain point with SQL - and then we'll see if it still needs work.

@donatj

This comment has been minimized.

Copy link

@donatj donatj commented Feb 12, 2020

I still believe a custom delimiter solution is ideal. A very limited HEREDOC-esq syntax such that you define your own delimiter and close with it would be ideal.

I know, I know

It was concluded that HEREDOC was not the correct syntax for Go to use

A big chunk of this conclusion came down to complicating features HEREDOC brings to other languages. None of these are inherent to all HEREDOC implementations though, and none of them would need to be implemented here. We don't need to cargo-cult them in.

The solution can be simple. It doesn't have to bring in every stupid feature other HEREDOC's have.

So hear me out, and then I'll slip back into the abyss.

Here's my rough proposal:

image

It fulfils most of what people are asking for above and on the previous PR:

  • Defining your own delimiter lets you more easily wrap any desired string content.
  • Very easy to READ exact content of string. No bytes between the delimiters that are not in the string other than leading and trailing newline
  • Very easy to paste an exact byte-for-byte document without having to reformat/rewrite in any way.

All this is to say maybe giving HEREDOC-esq syntax a second look would meet the actual needs of the community better and more cleanly than anything else being discussed here.

Yes, HEREDOC syntaxes can get complicated if you let them, but you don't have to let it get complicated.

If you just avoid the indentation options and other pitfalls other languages heredocs add complication with, it's fine and easy to read.

@MOZGIII

This comment has been minimized.

Copy link

@MOZGIII MOZGIII commented Feb 13, 2020

Btw, SQL use case, in particular, can be solved not via string, but via something like "include file" macro. It's kind of better in some sense, cause it's possible to compile it in such a way that the embedded file is in it's own section, and probably other smart things can be done to improve it. Of course, seeling the actual SQL along the code worth a lot more than some binary layout optimizations, so I'd vote for using the fixing the string. Although, maybe a modern editor can display the embedded file inline - then it becomes a to-go solution again...

@alanfo

This comment has been minimized.

Copy link

@alanfo alanfo commented Feb 13, 2020

Vis-à-vis HEREDOC style syntax, it's worth remembering that, even with the present raw string syntax, one can still just paste text straight in without further ado.

This is because in many of the contexts where raw strings are typically used a leading/trailing new line is unimportant though they can easily be trimmed off as the following example shows:

package main

import (
    "fmt"
    "strings"
)

func main() {
    s := `
Line 1
Line 2
Line 3
`   
    fmt.Printf("%d\n", s[0]) // 10 ('\n')

    t := strings.TrimSpace(`
Line 1
Line 2
Line 3
`)

    fmt.Printf("%d\n", t[0]) // 76 ('L')
}

Under my idea of just introducing """ as an alternative raw string delimiter to enable embedded back-ticks to be used, the position would be exactly the same.

Indentation is a problem for any approach though this is best dealt with IMO by adding a suitable function to the strings package rather than trying to do something within the language itself.

@alanfo

This comment has been minimized.

Copy link

@alanfo alanfo commented Feb 13, 2020

@MOZGIII

In my practice, it happened at least once, in a non-source-code context.

Can you remember the context(s) in which back-ticks and triple double quotes occurred in the same raw string?

I've been trying to gauge what we'd be missing by having a simple solution to this issue and, so far, the answer has been - not much.

@MOZGIII

This comment has been minimized.

Copy link

@MOZGIII MOZGIII commented Feb 13, 2020

It was very narrow case where that sequence was explicitly required to be displayed, unlikely someone else stumbles upon this. That said - I've had way more cases where backticks and/or triple quotes been isufficient for embedding code in them. They're not a solution for a generic case, that's a fact, but I've already said everything I had to say about this above.

@alanfo

This comment has been minimized.

Copy link

@alanfo alanfo commented Feb 13, 2020

So it appears that the only real difficulty is embedding source code in other languages and potentially in Go itself.

I wonder how common this is in the wild? In most cases, I suspect that the amount of source code would be such that you'd want to load it in from a file.

@MOZGIII

This comment has been minimized.

Copy link

@MOZGIII MOZGIII commented Feb 13, 2020

Almost every time I embed multi-line strings is to embed something that's code. 95%. I'd say it's a major use case. Very rarely I embed verbatim text paragraphs without typography (i.e. without html/markdown). And I have no idea what else can even be possibly considered not code.

@deanveloper

This comment has been minimized.

Copy link
Author

@deanveloper deanveloper commented Feb 13, 2020

I think an issue with having two separate delimiters to represent the same thing is that people may get curious as to what the difference between backticks and triple-quotes/heredocs/etc, especially if they have dealt with languages like Bash where every kind of quote means a different thing. Finding answers online can also often be difficult, as people may often have trouble putting their question into words, and there are always many answers, even if the answer in the end is simply "they mean the same thing".

So the huge advantage to delim`string`delim and ``string`` syntaxes is that they are similar to the existing raw string syntax, meaning that intuitively, they are no different from the single-backtick raw strings except for their delimiters.

@alanfo

This comment has been minimized.

Copy link

@alanfo alanfo commented Feb 13, 2020

Well, by source code, I meant code for general purpose programming languages as opposed to: SQL, XML, json, mark-up etc.

But, if embedding code is as common as @MOZGIII says it is, then the simple solution I was proposing is not really good enough.

FWIW, of the '100%' solutions I prefer the:

``"this is a `raw` "string" literal"``

idea which I think @ianlancetaylor came up with.

@alanfo

This comment has been minimized.

Copy link

@alanfo alanfo commented Feb 13, 2020

@deanveloper I don't really understand your point as ISTM that, no matter how you look at it, you're going to end up with two different types of delimiter for raw strings to solve the back-tick problem.

If one accepts that the problem is worth solving at language level, there are basically two questions to answer:

  1. Is having a constant alternative delimiter which is the same at both ends good enough to deal with nearly all of the cases which might occur in practice bearing in mind this approach can never deal with 100% of cases now matter how rare the chosen delimiter may be.

  2. If a constant delimiter isn't good enough, how variable does the second delimiter type need to be. The more variable it is the more difficult it is to learn, read, lex and describe in the spec.

As you will have gathered, I was hoping that 1. might be good enough but I'm now not so sure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.