New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: Go 2: spec: introduce structured tags #23637

Open
urandom opened this Issue Jan 31, 2018 · 33 comments

Comments

Projects
None yet
@urandom

urandom commented Jan 31, 2018

This proposal is for a new syntax for struct tags, one that is formally defined in the grammar and can be validated by the compiler.

Problem

The current struct tag format is defined in the spec as a string literal. It doesn't go into any detail of what the format of that string might look like. If the user somehow stumbles upon the reflect package, a simple space-separated, key:"value" convention is mentioned. It doesn't go into detail about what the value might be, since that format is at the discretion of the package that uses said tag. There will never be a tool that will help the user write the value of a tag, similarly to what gocode does with regular code. The format itself might be poorly documented, or hard to find, leading one to guess what can be put as a value. The reflect package itself is probably not the biggest user-facing package in the standard library as well, leading to a plethora of stackoverflow questions about how multiple tags can be specified. I myself have made the error a few times of using a comma to delimit the different tags.

Proposal

EDIT: the original proposal introduced a new type. After the initial discussion, it was decided that there is no need for a new type, as a struct type or custom types whose underlying types can be constant (string/numeric/bool/...) will do just as well.

A tag value can be either a struct, whose field types can be constant, or custom types, whose underlying types are constant. According to the go spec, that means a field/custom type can be either a string, a boolean, a rune, an integer, a floating-point, or a complex number. Example definition and usage:

package json

type Rules struct {
    Name string
    OmitEmpty bool
    Ignore bool
}

func processTags(f reflect.StructField) {
    // reflect.StructField.Tags []interface{}   
    for _ ,t := range f.Tags {
        if jt, ok := t.(Rules); ok {
              ...
              break
        }
    }
}
package sqlx

type Name string

Users can instantiate values of such types within struct definitions, surrounded by [ and ] and delimited by ,. The type cannot be omitted when the value is instantiated.

package mypackage

import json
import sqlx

type MyStruct struct {
      Value      string [json.Rules{Name: "value"}, sqlx.Name("value")]
      PrivateKey []byte [json.Rules{Ignore: true}]
}

Benefits

Tags are just types, they are clearly defined and are part of a package's types. Tools (such as gocode) may now be made for assisting in using such tags, reducing the cognitive burden on users. Package authors will not need to create "value" parsers for their supported tags. As a type, a tag is now a first-class citizen in godoc. Even if a tag lacks any kind of documentation, a user still has a fighting chance of using it, since they can now easily go to do definition of a tag and just look up its fields, or see the definition in godoc. Finally, if the user has misspelled something, the compiler will now inform them of an error, instead of it occurring either at runtime, or being silently ignored as is the case right now.

Side note

This proposal is strictly for replacing the current stuct tags. While the tag grammar can be extended to be applied to a lot more things that struct tags, this proposal is not suggesting that it should, and such a discussion should be done in a different proposal.

@gopherbot gopherbot added this to the Proposal milestone Jan 31, 2018

@gopherbot gopherbot added the Proposal label Jan 31, 2018

@ianlancetaylor

This comment has been minimized.

Show comment
Hide comment
@ianlancetaylor

ianlancetaylor Jan 31, 2018

Contributor

Related to #20165, which was recently declined. But this version is better, because it proposes an alternative.

Contributor

ianlancetaylor commented Jan 31, 2018

Related to #20165, which was recently declined. But this version is better, because it proposes an alternative.

@ianlancetaylor

This comment has been minimized.

Show comment
Hide comment
@ianlancetaylor

ianlancetaylor Jan 31, 2018

Contributor

I don't see any special need for a new tag type. You may as well simply say that a struct field may be followed by a comma separated list of values, and that those values are available via reflection on the struct.

On the other hand, something this proposal doesn't clearly address is that those values must be entirely computable at compile time. That is not a notion that the language currently defines for anything other than constants, and it would have to be carefully spelled out to decide what is permitted and what is not. For example, can a tag, under either the original definition or this new one, have a field of interface type?

Contributor

ianlancetaylor commented Jan 31, 2018

I don't see any special need for a new tag type. You may as well simply say that a struct field may be followed by a comma separated list of values, and that those values are available via reflection on the struct.

On the other hand, something this proposal doesn't clearly address is that those values must be entirely computable at compile time. That is not a notion that the language currently defines for anything other than constants, and it would have to be carefully spelled out to decide what is permitted and what is not. For example, can a tag, under either the original definition or this new one, have a field of interface type?

@urandom

This comment has been minimized.

Show comment
Hide comment
@urandom

urandom Jan 31, 2018

@ianlancetaylor
You raise an interesting point. A struct will pretty much have the same benefits as a new tag type would. I imagine it would probably make the implementation a bit simpler. Other types might only be useful if they are the underlying type of a custom one, and as such one would have to use them explicitly, otherwise there might be ambiguity when a constant is provided directly:

package sqlx

type ColumnName string
...

package main
import sqlx

type MyStruct struct {
    Total int64 [sqlx.ColumnName("total")]
}

vs what I would consider an invalid usage:


package main
import sqlx

type MyStruct struct {
    Total int64 ["total"]
}

For your second point, I assumed that it would be clear that any value for any field of a tag has to be a constant. Such a "restriction" makes it clear what can and cannot be a field type, and will rule out having a struct as a field type (or an interface, in your example).

urandom commented Jan 31, 2018

@ianlancetaylor
You raise an interesting point. A struct will pretty much have the same benefits as a new tag type would. I imagine it would probably make the implementation a bit simpler. Other types might only be useful if they are the underlying type of a custom one, and as such one would have to use them explicitly, otherwise there might be ambiguity when a constant is provided directly:

package sqlx

type ColumnName string
...

package main
import sqlx

type MyStruct struct {
    Total int64 [sqlx.ColumnName("total")]
}

vs what I would consider an invalid usage:


package main
import sqlx

type MyStruct struct {
    Total int64 ["total"]
}

For your second point, I assumed that it would be clear that any value for any field of a tag has to be a constant. Such a "restriction" makes it clear what can and cannot be a field type, and will rule out having a struct as a field type (or an interface, in your example).

@dlsniper

This comment has been minimized.

Show comment
Hide comment
@dlsniper

dlsniper Feb 5, 2018

Contributor

I wonder if we could solve this without having to change the language, and even better, in Go 1.X rather than waiting for Go 2. As such, I've tried to understand the problem as well as the proposed solution and came up with a different approach to the problem, please see below.

First, the problem.
I think the description starts from the wrong set of assumptions:

There will never be a tool that will help the user write the value of a tag, similarly to what gocode does with regular code.

There can totally be a tool that understands how these flags work and allow users to define custom tags and have them validated.
One such tool might for example benefit from "magic" comments in the code, for example, the structure proposed could be "annotated" with a comment like // +tag.

This would of course have the advantage of not having to force the change in the toolchain, with the downside that you'd need to have the tool to validate this instead of the compiler. The values should be json, for example:

package mypackage

import "json"
import "sqlx"

type MyStruct struct {
	Value string `json:"{\"Name\":\"value\"}" sqlx:"{\"Name\":\"value\"}"`
	PrivateKey []byte `json:"{\"Ignore\":true}"`
}
package json

// +tag
type Tag struct {
	Name string
	OmitEmpty bool
	Ignore bool
}
package sqlx

// +tag
type SQLXTag struct {
	Name string
}

More details can be put into this on how this should be a single tag per package, the struct must be exportable, and so on (which the current proposal also does not address).

The format itself might be poorly documented, or hard to find, leading one to guess what can be put as a value. The reflect package itself is probably not the biggest user-facing package in the standard library as well, leading to a plethora of stackoverflow questions about how multiple tags can be specified.

This sounds like a problem one could be able to fix with a CL / PR to the documentation of the package which specifically improves it by documenting the tag available or how to use these struct tags.

I myself have made the error a few times of using a comma to delimit the different tags.

Should the above proposal with "annotating" a struct in a package work, this means that the tools could also fix the problem of navigating to the definition of tag.

Furthermore, the original proposal adds the problem that tag now is a keyword and cannot be used as in regular code. Imho, should any new keyword be added in Go 2, there should be a really good reason to do so and it should be kept in mind that it would make the porting of existing Go 1 sources that much harder given how now the code needs to be refactored before being ported over.

The downside of my proposal is that this requires people to use non-compiler tools. But given how govet is now partially integrated in go test, this check could also be added to that list.

Tools that offer completion to users can be adapted to fulfill the requirement of assisting the user in writing the tag, all without having the language changed with an extra keyword added.

And should the compiler ever want to validate these tags, the code would already be there in govet,

Contributor

dlsniper commented Feb 5, 2018

I wonder if we could solve this without having to change the language, and even better, in Go 1.X rather than waiting for Go 2. As such, I've tried to understand the problem as well as the proposed solution and came up with a different approach to the problem, please see below.

First, the problem.
I think the description starts from the wrong set of assumptions:

There will never be a tool that will help the user write the value of a tag, similarly to what gocode does with regular code.

There can totally be a tool that understands how these flags work and allow users to define custom tags and have them validated.
One such tool might for example benefit from "magic" comments in the code, for example, the structure proposed could be "annotated" with a comment like // +tag.

This would of course have the advantage of not having to force the change in the toolchain, with the downside that you'd need to have the tool to validate this instead of the compiler. The values should be json, for example:

package mypackage

import "json"
import "sqlx"

type MyStruct struct {
	Value string `json:"{\"Name\":\"value\"}" sqlx:"{\"Name\":\"value\"}"`
	PrivateKey []byte `json:"{\"Ignore\":true}"`
}
package json

// +tag
type Tag struct {
	Name string
	OmitEmpty bool
	Ignore bool
}
package sqlx

// +tag
type SQLXTag struct {
	Name string
}

More details can be put into this on how this should be a single tag per package, the struct must be exportable, and so on (which the current proposal also does not address).

The format itself might be poorly documented, or hard to find, leading one to guess what can be put as a value. The reflect package itself is probably not the biggest user-facing package in the standard library as well, leading to a plethora of stackoverflow questions about how multiple tags can be specified.

This sounds like a problem one could be able to fix with a CL / PR to the documentation of the package which specifically improves it by documenting the tag available or how to use these struct tags.

I myself have made the error a few times of using a comma to delimit the different tags.

Should the above proposal with "annotating" a struct in a package work, this means that the tools could also fix the problem of navigating to the definition of tag.

Furthermore, the original proposal adds the problem that tag now is a keyword and cannot be used as in regular code. Imho, should any new keyword be added in Go 2, there should be a really good reason to do so and it should be kept in mind that it would make the porting of existing Go 1 sources that much harder given how now the code needs to be refactored before being ported over.

The downside of my proposal is that this requires people to use non-compiler tools. But given how govet is now partially integrated in go test, this check could also be added to that list.

Tools that offer completion to users can be adapted to fulfill the requirement of assisting the user in writing the tag, all without having the language changed with an extra keyword added.

And should the compiler ever want to validate these tags, the code would already be there in govet,

@creker

This comment has been minimized.

Show comment
Hide comment
@creker

creker Feb 5, 2018

@dlsniper

Furthermore, the original proposal adds the problem that tag now is a keyword and cannot be used as in regular code

You can always make compiler a bit smarter to understand context and when a keyword is a keyword. tag keyword would appear in a very specific context which compiler could easily detect and understand. No code could ever be broken with that new keyword. Other languages do that and have no problem adding new contextual keywords without breaking backwards compatibility.

As for your proposal, for me adding another set of magic comments further establishes opinion that there's something wrong with the design of the language. Every time I look at these comments they look out of place. Like someone forgot to add some feature and, in order to not break things, shoved everything into comments. There's plenty of magic comments already. I think we should stop and implement proper Go features and not continue developing another language on top of Go.

creker commented Feb 5, 2018

@dlsniper

Furthermore, the original proposal adds the problem that tag now is a keyword and cannot be used as in regular code

You can always make compiler a bit smarter to understand context and when a keyword is a keyword. tag keyword would appear in a very specific context which compiler could easily detect and understand. No code could ever be broken with that new keyword. Other languages do that and have no problem adding new contextual keywords without breaking backwards compatibility.

As for your proposal, for me adding another set of magic comments further establishes opinion that there's something wrong with the design of the language. Every time I look at these comments they look out of place. Like someone forgot to add some feature and, in order to not break things, shoved everything into comments. There's plenty of magic comments already. I think we should stop and implement proper Go features and not continue developing another language on top of Go.

@ianlancetaylor

This comment has been minimized.

Show comment
Hide comment
@ianlancetaylor

ianlancetaylor Feb 5, 2018

Contributor

You can always make compiler a bit smarter to understand context and when a keyword is a keyword.

As I said above, though, I see no advantage at all to using tag rather than struct.

This proposal still needs more clarity on precisely what is permitted in a tag type, whatever we call it. It's not enough to say "it has to be a constant." We need to spell out precisely what that means. Can it be a constant expression? What types are the fields permitted to have?

Contributor

ianlancetaylor commented Feb 5, 2018

You can always make compiler a bit smarter to understand context and when a keyword is a keyword.

As I said above, though, I see no advantage at all to using tag rather than struct.

This proposal still needs more clarity on precisely what is permitted in a tag type, whatever we call it. It's not enough to say "it has to be a constant." We need to spell out precisely what that means. Can it be a constant expression? What types are the fields permitted to have?

@urandom

This comment has been minimized.

Show comment
Hide comment
@urandom

urandom Feb 5, 2018

One such tool might for example benefit from "magic" comments in the code, for example, the structure proposed could be "annotated" with a comment like // +tag.

This seems to make the problem worse, to be honest. Instead of making tags easier to write and use, you are now introducing more magic comments. I'm sure I'm not the only one opposed to such solutions, as such comments are very confusing to users (plenty of questions on stackoverflow). Also, what happens when someone puts // +tag before multiple types?

Value string json:"{\"Name\":\"value\"}" sqlx:"{\"Name\":\"value\"}"

This not only ignores a major part of the problem, illustrated by the proposal (syntax), but also makes it harder to write.

More details can be put into this on how this should be a single tag per package, the struct must be exportable, and so on (which the current proposal also does not address).

Should we address the obvious? Honest question, I skipped some things as I thought they were too obvious to write.

This sounds like a problem one could be able to fix with a CL / PR to the documentation of the package which specifically improves it by documenting the tag available or how to use these struct tags.

It's still in the reflect package. Why would an average user ever go and read the reflect package. It's index alone is larger that the documentation of some packages.

Furthermore, the original proposal adds the problem that tag now is a keyword and cannot be used as in regular code. Imho, should any new keyword be added in Go 2, there should be a really good reason to do so and it should be kept in mind that it would make the porting of existing Go 1 sources that much harder given how now the code needs to be refactored before being ported over.

I edited my original proposal to remove the inclusion of a new type. This was discussed in the initial discussion with @ianlancetaylor, and I was hoping further discussion would include that as well.

urandom commented Feb 5, 2018

One such tool might for example benefit from "magic" comments in the code, for example, the structure proposed could be "annotated" with a comment like // +tag.

This seems to make the problem worse, to be honest. Instead of making tags easier to write and use, you are now introducing more magic comments. I'm sure I'm not the only one opposed to such solutions, as such comments are very confusing to users (plenty of questions on stackoverflow). Also, what happens when someone puts // +tag before multiple types?

Value string json:"{\"Name\":\"value\"}" sqlx:"{\"Name\":\"value\"}"

This not only ignores a major part of the problem, illustrated by the proposal (syntax), but also makes it harder to write.

More details can be put into this on how this should be a single tag per package, the struct must be exportable, and so on (which the current proposal also does not address).

Should we address the obvious? Honest question, I skipped some things as I thought they were too obvious to write.

This sounds like a problem one could be able to fix with a CL / PR to the documentation of the package which specifically improves it by documenting the tag available or how to use these struct tags.

It's still in the reflect package. Why would an average user ever go and read the reflect package. It's index alone is larger that the documentation of some packages.

Furthermore, the original proposal adds the problem that tag now is a keyword and cannot be used as in regular code. Imho, should any new keyword be added in Go 2, there should be a really good reason to do so and it should be kept in mind that it would make the porting of existing Go 1 sources that much harder given how now the code needs to be refactored before being ported over.

I edited my original proposal to remove the inclusion of a new type. This was discussed in the initial discussion with @ianlancetaylor, and I was hoping further discussion would include that as well.

@urandom

This comment has been minimized.

Show comment
Hide comment
@urandom

urandom Feb 5, 2018

This proposal still needs more clarity on precisely what is permitted in a tag type, whatever we call it. It's not enough to say "it has to be a constant." We need to spell out precisely what that means. Can it be a constant expression? What types are the fields permitted to have?

I've edited the proposal to add more information as to what types are permitted as tags.

urandom commented Feb 5, 2018

This proposal still needs more clarity on precisely what is permitted in a tag type, whatever we call it. It's not enough to say "it has to be a constant." We need to spell out precisely what that means. Can it be a constant expression? What types are the fields permitted to have?

I've edited the proposal to add more information as to what types are permitted as tags.

@giniedp

This comment has been minimized.

Show comment
Hide comment
@giniedp

giniedp Feb 5, 2018

i like the idea of the proposal. I would love to be able to write Metadata that is then checked at compile time, and that i do not need to parse the metadata at runtime.

The updated proposal makes sense to me. A Metadata is simply a struct.
Also the syntax is ok for the tags. It event lets me write the metadata line by line

1. initial proposal

type MyStruct struct {
    Value string [
        json.Rules{Name: "value"}, 
        sqlx.Name{"value"}
    ]
    PrivateKey []byte [
        json.Rules{Ignore: true}
    ]
}

That appears even to be readable (at least to me). Fields and annotations are clearly distinguishable, even without syntax highlighting.

Now, i know this is about tags only but if we would want to add metadata to other things than struct fields, i would suggest to put the metadata above the thing that you annotate instead of having it behind.

2 examples that arise to me, are C# Attributes and Java Annotations. Lets see how they would look like on go struct fields

2. C# like

type MyStruct struct {
    [json.Rules{Name: "value"}]
    [sqlx.Name{"value"}]
    Value string

    [json.Rules{Ignore: true}]
    PrivateKey []byte 
}

That is less readable than 1.

3. Java like

type MyStruct struct {
    @json.Rules{Name: "value"}
    @sqlx.Name{"value"}
    Value string

    @json.Rules{Ignore: true}
    PrivateKey []byte 
}

That is less readable than 1. but way cleaner than 2.

Now in go we already have a syntax for multiple imports and multiple constants. Lets try that

4. Go like

type MyStruct struct {
    meta (
        json.Rules{Name: "value"}
        sqlx.Name{"value"}
    )
    Value string

    meta (json.Rules{Ignore: true})
    PrivateKey []byte 
}

That is less compact. Removing the meta keyword wouldnt help i think. Neither when using square brackets

5. with square brackets

type MyStruct struct {
    [
        json.Rules{Name: "value"}
        sqlx.Name{"value"}
    ]
    Value string

    [json.Rules{Ignore: true}]
    PrivateKey []byte 
}

The single statement looks like 2. C# like and the multiline statement is still not as compact as i would like it to be.

So far i still like the 3. Java like style best. However, if metadata should not be applied to anything other than struct fields (ever) then i prefer the 1. initial proposal style. Now if there are some legal issues with stealing a syntax from another language (i am not a lawyer) then i could think of following

6. hash

type MyStruct struct {
    # json.Rules{Name: "value"}
    # sqlx.Name{"value"}
    Value string

    # json.Rules{Ignore: true}
    PrivateKey []byte 
}

giniedp commented Feb 5, 2018

i like the idea of the proposal. I would love to be able to write Metadata that is then checked at compile time, and that i do not need to parse the metadata at runtime.

The updated proposal makes sense to me. A Metadata is simply a struct.
Also the syntax is ok for the tags. It event lets me write the metadata line by line

1. initial proposal

type MyStruct struct {
    Value string [
        json.Rules{Name: "value"}, 
        sqlx.Name{"value"}
    ]
    PrivateKey []byte [
        json.Rules{Ignore: true}
    ]
}

That appears even to be readable (at least to me). Fields and annotations are clearly distinguishable, even without syntax highlighting.

Now, i know this is about tags only but if we would want to add metadata to other things than struct fields, i would suggest to put the metadata above the thing that you annotate instead of having it behind.

2 examples that arise to me, are C# Attributes and Java Annotations. Lets see how they would look like on go struct fields

2. C# like

type MyStruct struct {
    [json.Rules{Name: "value"}]
    [sqlx.Name{"value"}]
    Value string

    [json.Rules{Ignore: true}]
    PrivateKey []byte 
}

That is less readable than 1.

3. Java like

type MyStruct struct {
    @json.Rules{Name: "value"}
    @sqlx.Name{"value"}
    Value string

    @json.Rules{Ignore: true}
    PrivateKey []byte 
}

That is less readable than 1. but way cleaner than 2.

Now in go we already have a syntax for multiple imports and multiple constants. Lets try that

4. Go like

type MyStruct struct {
    meta (
        json.Rules{Name: "value"}
        sqlx.Name{"value"}
    )
    Value string

    meta (json.Rules{Ignore: true})
    PrivateKey []byte 
}

That is less compact. Removing the meta keyword wouldnt help i think. Neither when using square brackets

5. with square brackets

type MyStruct struct {
    [
        json.Rules{Name: "value"}
        sqlx.Name{"value"}
    ]
    Value string

    [json.Rules{Ignore: true}]
    PrivateKey []byte 
}

The single statement looks like 2. C# like and the multiline statement is still not as compact as i would like it to be.

So far i still like the 3. Java like style best. However, if metadata should not be applied to anything other than struct fields (ever) then i prefer the 1. initial proposal style. Now if there are some legal issues with stealing a syntax from another language (i am not a lawyer) then i could think of following

6. hash

type MyStruct struct {
    # json.Rules{Name: "value"}
    # sqlx.Name{"value"}
    Value string

    # json.Rules{Ignore: true}
    PrivateKey []byte 
}
@urandom

This comment has been minimized.

Show comment
Hide comment
@urandom

urandom Feb 5, 2018

Having the field tags before the field declaration is not very readable, compared to having them afterwards. For the same reason that it is more readable to have the type of a variable after the variable. When you start reading the struct definition, you come upon some meta information about something you haven't yet read. Currently, you know that a PrimaryKey is a byte array, and it is ignored for json marshaling. With your suggestioun, You know that something will be ignored for json marshaling, and afterwards you learn that what is ignored will be a primary key, which is a byte array.

urandom commented Feb 5, 2018

Having the field tags before the field declaration is not very readable, compared to having them afterwards. For the same reason that it is more readable to have the type of a variable after the variable. When you start reading the struct definition, you come upon some meta information about something you haven't yet read. Currently, you know that a PrimaryKey is a byte array, and it is ignored for json marshaling. With your suggestioun, You know that something will be ignored for json marshaling, and afterwards you learn that what is ignored will be a primary key, which is a byte array.

@giniedp

This comment has been minimized.

Show comment
Hide comment
@giniedp

giniedp Feb 5, 2018

I understand your point and i partially agree on that. Having metadata on tags only, your suggestion looks best (i might want to omit the comma in multiline though)

My intention is to suggest a syntax that might work on structs and methods. Those elements have their own body. Adding metadata behind the body might push it out of sight if the implementation is lengthy. I think metadata should live somewhere near to the name of the element that it annotates and my natural choice is above that name, since below is the body.

Syntax highlighting helps to spot the parts of the code you are interested in. So if you are interested in reading the struct definition, your eye will skip the metadata syntax.

giniedp commented Feb 5, 2018

I understand your point and i partially agree on that. Having metadata on tags only, your suggestion looks best (i might want to omit the comma in multiline though)

My intention is to suggest a syntax that might work on structs and methods. Those elements have their own body. Adding metadata behind the body might push it out of sight if the implementation is lengthy. I think metadata should live somewhere near to the name of the element that it annotates and my natural choice is above that name, since below is the body.

Syntax highlighting helps to spot the parts of the code you are interested in. So if you are interested in reading the struct definition, your eye will skip the metadata syntax.

@creker

This comment has been minimized.

Show comment
Hide comment
@creker

creker Feb 5, 2018

I don't think tags are that important to care about them being out of sight. For the most part, I only care about actual fields and look at tags only in very specific cases. It's actually a good thing that they're out of the way because most of the time you don't need to look at them.

Your examples with @ and # prefixes look good and readable but I don't think it's that important to pursue and change existing syntax. Even C# syntax is easy to read for me being a C# programmer.

creker commented Feb 5, 2018

I don't think tags are that important to care about them being out of sight. For the most part, I only care about actual fields and look at tags only in very specific cases. It's actually a good thing that they're out of the way because most of the time you don't need to look at them.

Your examples with @ and # prefixes look good and readable but I don't think it's that important to pursue and change existing syntax. Even C# syntax is easy to read for me being a C# programmer.

@urandom

This comment has been minimized.

Show comment
Hide comment
@urandom

urandom Feb 12, 2018

Just wanted to add another quick anecdote. I recently saw this committed by a colleague of mine, a seasoned developer:

ID int `json: "id"`

Obviously this wasn't tested at runtime, but its clear that even the best of us can overlook the syntax, especially since were are used to catching 'syntax' errors at compile time.

urandom commented Feb 12, 2018

Just wanted to add another quick anecdote. I recently saw this committed by a colleague of mine, a seasoned developer:

ID int `json: "id"`

Obviously this wasn't tested at runtime, but its clear that even the best of us can overlook the syntax, especially since were are used to catching 'syntax' errors at compile time.

@kprav33n

This comment has been minimized.

Show comment
Hide comment
@kprav33n

kprav33n Feb 12, 2018

I like this proposal for property tags. Would it be too much to ask for a similar feature for struct-level tags?

kprav33n commented Feb 12, 2018

I like this proposal for property tags. Would it be too much to ask for a similar feature for struct-level tags?

@ianlancetaylor

This comment has been minimized.

Show comment
Hide comment
@ianlancetaylor

ianlancetaylor Feb 12, 2018

Contributor

@kprav33n I'm not sure what you mean by struct-level tags, but I'm guessing it's something that the language does not support today. It sounds like an orthogonal idea that should be discussed separately from this issue.

Contributor

ianlancetaylor commented Feb 12, 2018

@kprav33n I'm not sure what you mean by struct-level tags, but I'm guessing it's something that the language does not support today. It sounds like an orthogonal idea that should be discussed separately from this issue.

@kprav33n

This comment has been minimized.

Show comment
Hide comment
@kprav33n

kprav33n Feb 13, 2018

@ianlancetaylor Thanks for the comment. Yes, this feature doesn't exist in the language today. Will discuss as a separate issue.

kprav33n commented Feb 13, 2018

@ianlancetaylor Thanks for the comment. Yes, this feature doesn't exist in the language today. Will discuss as a separate issue.

@Backfighter

This comment has been minimized.

Show comment
Hide comment
@Backfighter

Backfighter Mar 25, 2018

I really like this proposal since I encountered problems with the current implementation of tags

type MyStruct struct {
    field string `tag:"This is an extremely long tag and it's hard to view on smaller screens since it is so incredibly long, but it can't be broken using a line break because that would prevent correct parsing of the tag."`
}

Adding line breaks will prevent the tag from being parsed correctly:

package main

import (
	"fmt"
	"reflect"
)

type MyStruct struct {
	field string `tag:"This is an extremely long tag and it's hard to view on smaller 
screens since it is so incredibly long, but it can't be broken using a 
line break because that would prevent correct parsing of the tag."`
}

func main() {
	v, ok := reflect.ValueOf(MyStruct{}).Type().Field(0).Tag.Lookup("tag")
	fmt.Printf("%q, %t", v, ok)
	// Output: "", false
}

This is kind of documented at https://golang.org/pkg/reflect/#StructTag which clearly forbids line breaks between the "tag-pairs" and says that the quoted string part is in "Go string literal syntax". Which means "interpreted string literal" per specification.
In this case a compile time check could have saved me some debugging time.

Backfighter commented Mar 25, 2018

I really like this proposal since I encountered problems with the current implementation of tags

type MyStruct struct {
    field string `tag:"This is an extremely long tag and it's hard to view on smaller screens since it is so incredibly long, but it can't be broken using a line break because that would prevent correct parsing of the tag."`
}

Adding line breaks will prevent the tag from being parsed correctly:

package main

import (
	"fmt"
	"reflect"
)

type MyStruct struct {
	field string `tag:"This is an extremely long tag and it's hard to view on smaller 
screens since it is so incredibly long, but it can't be broken using a 
line break because that would prevent correct parsing of the tag."`
}

func main() {
	v, ok := reflect.ValueOf(MyStruct{}).Type().Field(0).Tag.Lookup("tag")
	fmt.Printf("%q, %t", v, ok)
	// Output: "", false
}

This is kind of documented at https://golang.org/pkg/reflect/#StructTag which clearly forbids line breaks between the "tag-pairs" and says that the quoted string part is in "Go string literal syntax". Which means "interpreted string literal" per specification.
In this case a compile time check could have saved me some debugging time.

@jimmyfrasche

This comment has been minimized.

Show comment
Hide comment
@jimmyfrasche

jimmyfrasche Mar 25, 2018

Member

I'm not sure if this is worth the increase in complexity to the language, etc., etc. But the potential benefits to tooling and improvement to developer experience do seem quite nice, though. I think it is worth exploring the idea.

I'm not concerned about the particulars of the syntax, so I'll stick with the convention in the first post. (To give the [] syntax a distinct name, I'll call it a vector.)

There's discussion about extending what kinds of types can be constants—which, currently, seems to be mostly taking place in #21130. Let's assume, for the moment that, it's extended to allow a struct whose fields are all types that can be constants.

While I agree that a tag should be a defined type, I don't think that should be enforced by the compiler—that can be left to linters.

With the above, the proposal reduces to: any vector of constants is a valid tag.

This also allows an old-style tag to be easily converted to a new-style tag.

For example, say we have some struct with a field like

Field Type `json:",omitempty" something:"else" another:"thing"`

Given a tool with a complete understanding of the tags defined in the stdlib but no third party libraries, this could be automatically rewritten to

Field Type [
    json.Rules{OmityEmpty: true},
    `something:"else"`,
    `another:"thing"`,
]

Then, the third party tags could be manually rewritten or rewritten further by tools provided by the third party packages.

It would also be possible for the reflect API to work with both old- and new-style tags: Get and Lookup would search for a tag that is an untyped string with the old-style format in the vector of constants while a new API allowed introspection of the new-style tags.

I'd also note that most of the benefits of this proposal are for tooling and increased compile time checking. There's little benefit for the program at runtime, but there are some:

  1. No parser needed in already reflect-heavy code, reducing the bug/security surface while requiring fewer tests
  2. Tags can have methods, potentially allowing a better factoring of the code for handling the tags.
Member

jimmyfrasche commented Mar 25, 2018

I'm not sure if this is worth the increase in complexity to the language, etc., etc. But the potential benefits to tooling and improvement to developer experience do seem quite nice, though. I think it is worth exploring the idea.

I'm not concerned about the particulars of the syntax, so I'll stick with the convention in the first post. (To give the [] syntax a distinct name, I'll call it a vector.)

There's discussion about extending what kinds of types can be constants—which, currently, seems to be mostly taking place in #21130. Let's assume, for the moment that, it's extended to allow a struct whose fields are all types that can be constants.

While I agree that a tag should be a defined type, I don't think that should be enforced by the compiler—that can be left to linters.

With the above, the proposal reduces to: any vector of constants is a valid tag.

This also allows an old-style tag to be easily converted to a new-style tag.

For example, say we have some struct with a field like

Field Type `json:",omitempty" something:"else" another:"thing"`

Given a tool with a complete understanding of the tags defined in the stdlib but no third party libraries, this could be automatically rewritten to

Field Type [
    json.Rules{OmityEmpty: true},
    `something:"else"`,
    `another:"thing"`,
]

Then, the third party tags could be manually rewritten or rewritten further by tools provided by the third party packages.

It would also be possible for the reflect API to work with both old- and new-style tags: Get and Lookup would search for a tag that is an untyped string with the old-style format in the vector of constants while a new API allowed introspection of the new-style tags.

I'd also note that most of the benefits of this proposal are for tooling and increased compile time checking. There's little benefit for the program at runtime, but there are some:

  1. No parser needed in already reflect-heavy code, reducing the bug/security surface while requiring fewer tests
  2. Tags can have methods, potentially allowing a better factoring of the code for handling the tags.
@jimmyfrasche

This comment has been minimized.

Show comment
Hide comment
@jimmyfrasche

jimmyfrasche Apr 20, 2018

Member

Some points brought up in #24889 by myself, @ianlancetaylor, @balasanjay, and @bcmills

If the tags are any allowed constant they could also be named constants and not just constant literals, for example:

const ComplicatedTag = pkg.Qual{ /* a lot of fields */ }
type S struct {
  A int [ ComplicatedTag ]
  B string [ ComplicatedTag, pkg.Other("tag") ]
  C interface{} [ ComplicatedTag ]
}

which allows both reuse and documentation on ComplicatedTag

Tags, being ordinary types, can use versioning which in turn allows them to be standardized and collected in central locations.

Member

jimmyfrasche commented Apr 20, 2018

Some points brought up in #24889 by myself, @ianlancetaylor, @balasanjay, and @bcmills

If the tags are any allowed constant they could also be named constants and not just constant literals, for example:

const ComplicatedTag = pkg.Qual{ /* a lot of fields */ }
type S struct {
  A int [ ComplicatedTag ]
  B string [ ComplicatedTag, pkg.Other("tag") ]
  C interface{} [ ComplicatedTag ]
}

which allows both reuse and documentation on ComplicatedTag

Tags, being ordinary types, can use versioning which in turn allows them to be standardized and collected in central locations.

@leafbebop

This comment has been minimized.

Show comment
Hide comment
@leafbebop

leafbebop Jul 11, 2018

I dislike the idea of binding tags and external packages. While field tags are very widely used by those external packages, binding them is very hindering.

Having an json:... tag does not means the struct (or the package holding the struct) should have a dependency on encoding/json: since that Go currently does not have ways to modify field tags, it makes sense to be there for other packages (usually in the same program) to marshal/unmarshal. Being dependent on encoding/json does not make sense.

I think field tags have problems need to be addressed, like OP said: It needs syntax check and tools helping that. But binding them to dependency feels overdoing.

leafbebop commented Jul 11, 2018

I dislike the idea of binding tags and external packages. While field tags are very widely used by those external packages, binding them is very hindering.

Having an json:... tag does not means the struct (or the package holding the struct) should have a dependency on encoding/json: since that Go currently does not have ways to modify field tags, it makes sense to be there for other packages (usually in the same program) to marshal/unmarshal. Being dependent on encoding/json does not make sense.

I think field tags have problems need to be addressed, like OP said: It needs syntax check and tools helping that. But binding them to dependency feels overdoing.

@urandom

This comment has been minimized.

Show comment
Hide comment
@urandom

urandom Jul 11, 2018

@leafbebop

Why would importing the json package pose problems? Or in fact, any package that will provide tags? The only problem with dependencies is circular imports, which would never happen here. And so far, I haven't seen that many third-party packages that use other third party packages' tag definitions, which means that more or less all current tags, like json, are pretty much placing a dependency on a package that defines them, since you are also more than likely to be importing said package somewhere else in your code in order to work with the structs that have these tags.

urandom commented Jul 11, 2018

@leafbebop

Why would importing the json package pose problems? Or in fact, any package that will provide tags? The only problem with dependencies is circular imports, which would never happen here. And so far, I haven't seen that many third-party packages that use other third party packages' tag definitions, which means that more or less all current tags, like json, are pretty much placing a dependency on a package that defines them, since you are also more than likely to be importing said package somewhere else in your code in order to work with the structs that have these tags.

@leafbebop

This comment has been minimized.

Show comment
Hide comment
@leafbebop

leafbebop Jul 11, 2018

@urandom

Say I am writing a website, and since Go is supporting Wasm, I am going fullstack. Because of that, I isolate my code of data models for re-usabilty.

Now that there is not a way to add field tags to struct, for my server code to be able to co-operate with sql, I add tag fields for sqlx. To do so, I imported sqlx, of course.

And then I decide to re-use the code of data models for my Wasm-based frontend, to trully enjoy the benefit of full-stacking. But here is the problem. I imported sqlx, and sqlx has an init function, which means the whole package cannot be eleminated by DCE - that means binary size increases for no gain at all - and for Wasm, binary size is critical. The worst part is yet to come: sqlx uses cgo, which cannot be compiled to Wasm (yet, but I do not think Go will ever come to a point that compiles C to Wasm, that's pure magic).

Sure, I can just copy my code, delete all tags and build it again. But why should I? The code suddenly becomes non-cross-platform-able (now JS is a GOOS) just because something trivial. It does not make sence.

Alternatively, I think it can remains in a keyword style - instead of package, use a string-typed keyword.

leafbebop commented Jul 11, 2018

@urandom

Say I am writing a website, and since Go is supporting Wasm, I am going fullstack. Because of that, I isolate my code of data models for re-usabilty.

Now that there is not a way to add field tags to struct, for my server code to be able to co-operate with sql, I add tag fields for sqlx. To do so, I imported sqlx, of course.

And then I decide to re-use the code of data models for my Wasm-based frontend, to trully enjoy the benefit of full-stacking. But here is the problem. I imported sqlx, and sqlx has an init function, which means the whole package cannot be eleminated by DCE - that means binary size increases for no gain at all - and for Wasm, binary size is critical. The worst part is yet to come: sqlx uses cgo, which cannot be compiled to Wasm (yet, but I do not think Go will ever come to a point that compiles C to Wasm, that's pure magic).

Sure, I can just copy my code, delete all tags and build it again. But why should I? The code suddenly becomes non-cross-platform-able (now JS is a GOOS) just because something trivial. It does not make sence.

Alternatively, I think it can remains in a keyword style - instead of package, use a string-typed keyword.

@urandom

This comment has been minimized.

Show comment
Hide comment
@urandom

urandom Jul 11, 2018

@leafbebop
I'd say that first of all, that is an incredibly specific example with not a lot of real world consequence. That being said, since you are worried about the resulting file size (again, not something a lot of us care about on a daily basis), you can also copy the struct and omit the tags. As you said so yourself, that would work just fine.

Such code would be in the extreme minority. Not only would it target wasm, but would also have to import cgo modules. Not a lot of people will have to deal with that. And why should everyone else have to suffer the current error-prone structureless mess that you have to triple check when you write, because no tool will be able to help you with that, and then pray that down the line you won't get any runtime errors because a field didn't match?

urandom commented Jul 11, 2018

@leafbebop
I'd say that first of all, that is an incredibly specific example with not a lot of real world consequence. That being said, since you are worried about the resulting file size (again, not something a lot of us care about on a daily basis), you can also copy the struct and omit the tags. As you said so yourself, that would work just fine.

Such code would be in the extreme minority. Not only would it target wasm, but would also have to import cgo modules. Not a lot of people will have to deal with that. And why should everyone else have to suffer the current error-prone structureless mess that you have to triple check when you write, because no tool will be able to help you with that, and then pray that down the line you won't get any runtime errors because a field didn't match?

@leafbebop

This comment has been minimized.

Show comment
Hide comment
@leafbebop

leafbebop Jul 11, 2018

@urandom

No. The problem here is just: Data modeling should not be depend on an external package, by theory or by practise. And asking a dependency because of a piece of meta info that may or may not be used, is, non-idomatic and does not feel likes Go.

You don't need to import io to implent an io.Reader but you need to import sqlx to define a data model that might be used by sqlx? It seems wrong to me.

And about error-prone part. Detecting errors before running is what Go is good at. But not all those detecting happens when you run go build. There are many other tools, including go vet, to check things like that and, as far as I am concerned, a Go tool is not hard to write.

I am not against to have a better meta-info (be it tag or not) about fields, because the old way is not expressive, and hard for tools to check. But binding a package for it? That is another problem.

What I propose as keyword style, is that somehow we have a syntax like:

type S struct {
    F T meta (
        "json" {
            omitempty bool = true
        }
    )
}

P.S.: I don't think that re-using code between front-end and back-end, especially data model code is rare; And from what I read, Go with Wasm is widely welcomed and far from minority. But those are beyond the scope of this issue.

leafbebop commented Jul 11, 2018

@urandom

No. The problem here is just: Data modeling should not be depend on an external package, by theory or by practise. And asking a dependency because of a piece of meta info that may or may not be used, is, non-idomatic and does not feel likes Go.

You don't need to import io to implent an io.Reader but you need to import sqlx to define a data model that might be used by sqlx? It seems wrong to me.

And about error-prone part. Detecting errors before running is what Go is good at. But not all those detecting happens when you run go build. There are many other tools, including go vet, to check things like that and, as far as I am concerned, a Go tool is not hard to write.

I am not against to have a better meta-info (be it tag or not) about fields, because the old way is not expressive, and hard for tools to check. But binding a package for it? That is another problem.

What I propose as keyword style, is that somehow we have a syntax like:

type S struct {
    F T meta (
        "json" {
            omitempty bool = true
        }
    )
}

P.S.: I don't think that re-using code between front-end and back-end, especially data model code is rare; And from what I read, Go with Wasm is widely welcomed and far from minority. But those are beyond the scope of this issue.

@jimmyfrasche

This comment has been minimized.

Show comment
Hide comment
@jimmyfrasche

jimmyfrasche Jul 11, 2018

Member

@leafbebop Alternately, it could just be best practice to put the tag structs in a separate package when using init to avoid the coupling.

You don't need to import io to define an io.Reader, but you do need to import time to have a time.Duration field, which seems the more apt analogy here.

Member

jimmyfrasche commented Jul 11, 2018

@leafbebop Alternately, it could just be best practice to put the tag structs in a separate package when using init to avoid the coupling.

You don't need to import io to define an io.Reader, but you do need to import time to have a time.Duration field, which seems the more apt analogy here.

@leafbebop

This comment has been minimized.

Show comment
Hide comment
@leafbebop

leafbebop Jul 11, 2018

@jimmyfrasche That requires rewrites to all packages using database/sql (and image). Which does not seem best or good to me. And furthermore, it does make sense.

Field tags and field types are very different on aspect of code logic. Field types are determinant on how the struct is organized and how the logic of that struct is written. On the other hand, field tags are, descriptive info about that field, often offered to other package.

That means, when you declare a field is a time.Duration, the type - the data structure hold a time.Duration and logic of that type use time.Duration. But if you have a field tag as json:omitempty, its logic can often has absolutely nothing with json. There is a reason why current spec allows assigning structs with different tags.

On the other hand, field tags are more like interface: They are both about how "outside" code use the type. An io.Writer does not care about how data to write is produced as long as it is a []byte, as a field with yaml: name does not care how and by which package (yaml has multiple non-official package to parse) any value is unmarshalled into that field as long as its name is name.

leafbebop commented Jul 11, 2018

@jimmyfrasche That requires rewrites to all packages using database/sql (and image). Which does not seem best or good to me. And furthermore, it does make sense.

Field tags and field types are very different on aspect of code logic. Field types are determinant on how the struct is organized and how the logic of that struct is written. On the other hand, field tags are, descriptive info about that field, often offered to other package.

That means, when you declare a field is a time.Duration, the type - the data structure hold a time.Duration and logic of that type use time.Duration. But if you have a field tag as json:omitempty, its logic can often has absolutely nothing with json. There is a reason why current spec allows assigning structs with different tags.

On the other hand, field tags are more like interface: They are both about how "outside" code use the type. An io.Writer does not care about how data to write is produced as long as it is a []byte, as a field with yaml: name does not care how and by which package (yaml has multiple non-official package to parse) any value is unmarshalled into that field as long as its name is name.

@jimmyfrasche

This comment has been minimized.

Show comment
Hide comment
@jimmyfrasche

jimmyfrasche Jul 11, 2018

Member

@leafbebop Why would packages using database/sql or image need to be rewritten? They use init but they don't expose any struct tags (unless I missed something re-skimming their docs to double check).

sqlx might need a separate package to define the types to use for the tags it provides, but it would need to define those types somewhere as part of the transition, regardless, so it would mostly just be a matter of what directory the file containing those definitions goes in.

I do get the yaml problem, though. It would be bad if every yaml parser was dependent on every other yaml parser just so that they could understand each others tags. Of course, it would also be possible for them to work together on a central repository that just contains the least-common-denominator tag definitions that they all agree on. They might need to define additional tag types locally but those could be upstreamed to the central repo later. This would have been an unmanageable mess before type aliases and vgo, but it no longer seems like it would be much of an issue. The tags being ordinary types allows them to use these existing mechanisms.

Member

jimmyfrasche commented Jul 11, 2018

@leafbebop Why would packages using database/sql or image need to be rewritten? They use init but they don't expose any struct tags (unless I missed something re-skimming their docs to double check).

sqlx might need a separate package to define the types to use for the tags it provides, but it would need to define those types somewhere as part of the transition, regardless, so it would mostly just be a matter of what directory the file containing those definitions goes in.

I do get the yaml problem, though. It would be bad if every yaml parser was dependent on every other yaml parser just so that they could understand each others tags. Of course, it would also be possible for them to work together on a central repository that just contains the least-common-denominator tag definitions that they all agree on. They might need to define additional tag types locally but those could be upstreamed to the central repo later. This would have been an unmanageable mess before type aliases and vgo, but it no longer seems like it would be much of an issue. The tags being ordinary types allows them to use these existing mechanisms.

@leafbebop

This comment has been minimized.

Show comment
Hide comment
@leafbebop

leafbebop Jul 12, 2018

@jimmyfrasche I think you missed my points so I summarized it here:

  1. By philosophy, field tags are descriptive pieces of info. It is oriented to outer program. Much like implenmenting an io.Reader is not specified to be used in package io, having a json tag does not necessarily means the model is for json. It is wrong on philosophy to bind such information to a concrete use, let alone a specific package.

  2. By theory, even when a field is meant for a certain topic (json,sql,or yaml and so on), that topic does not bind to a certain package. Though versioning and type alias might simplify a unified tag system, but that seems complicated and non-idom for no reason. As I put it, it is literally just a topic and to adrress a general topic, a keyword is clear, to the point and easy to implement.

  3. By practise, having extra dependency on a data model package can cause problem. Every single sql driver has an init function (to utilize database/sql) and if it has a tag to interept, it means overhead.

And there is really no drawback for using a keyword-styled structured tag. The old form of field tag spec has two aspects being weak. Those are, expresiveness and error checking.

The expressiveness problem is solved in almost identical way of a "package" way, so I'll leave it there. And as I said in previous comment, error checking can happen outside of compile. It is not hard at all to write a go tool that checks field tags, and it can easily scale well to custom schema.

leafbebop commented Jul 12, 2018

@jimmyfrasche I think you missed my points so I summarized it here:

  1. By philosophy, field tags are descriptive pieces of info. It is oriented to outer program. Much like implenmenting an io.Reader is not specified to be used in package io, having a json tag does not necessarily means the model is for json. It is wrong on philosophy to bind such information to a concrete use, let alone a specific package.

  2. By theory, even when a field is meant for a certain topic (json,sql,or yaml and so on), that topic does not bind to a certain package. Though versioning and type alias might simplify a unified tag system, but that seems complicated and non-idom for no reason. As I put it, it is literally just a topic and to adrress a general topic, a keyword is clear, to the point and easy to implement.

  3. By practise, having extra dependency on a data model package can cause problem. Every single sql driver has an init function (to utilize database/sql) and if it has a tag to interept, it means overhead.

And there is really no drawback for using a keyword-styled structured tag. The old form of field tag spec has two aspects being weak. Those are, expresiveness and error checking.

The expressiveness problem is solved in almost identical way of a "package" way, so I'll leave it there. And as I said in previous comment, error checking can happen outside of compile. It is not hard at all to write a go tool that checks field tags, and it can easily scale well to custom schema.

@balasanjay

This comment has been minimized.

Show comment
Hide comment
@balasanjay

balasanjay Jul 12, 2018

Contributor

@leafbebop That counter-proposal seems to lack a significant amount of detail that makes it hard to evaluate; you should consider opening a separate proposal, if you think that is the way to go. Among other things, I don't understand how such "error checking" could be implemented, nor do I understand how packages would read that data (e.g. how would the json package read whether a field is tagged with omit empty). If the binding here are just strings, then how would we globally agree on who gets the "sql" string, or the "json" string, when validating a struct with a particular tag? There are multiple packages that deal with these subjects, and they may well want different data structures. Its also unclear how to version changes to these structures.

To discuss your objections:

  1. I agree that field tags are descriptive pieces of info, oriented to an "outer" program. But I don't understand how it follows that its philosophically wrong to bind the information to a concrete use. I also see little difference whether you say "json" as a string, or json as a package reference (which you import as a string "encoding/json"). In either case, you're clearly describing the exact same thing semantically. (And here, there is no init function, nor any loss in dead-code-elimination).

  2. Versioning and type aliases were presumably referenced to discuss how a change like this (or any approach which namespaces symbols) enables evolution in the ecosystem, using the same tools that we use for evolving structs. And until you address how your counterproposal would deal with that problem, it is hard to evaluate its effectiveness.

  3. Let's say that we grant that this use-case is a valid one[1]; to me, it seems that its a straightforward anti-pattern to define tag-structs in packages that have init functions (or call cgo, or use unsafe, or include a giant dependency tree). First, the language does not need to prevent every anti-pattern, it can't. Second, consider if you're the author of such a sql package and you get a bug report to this effect ("I'd like to use the tag without pulling in the whole DB driver"); it is rather straightforward to fix in a completely backwards-compatible manner. Simply add a new "sqltag" package (or whatever you want to call it), move the tag types in there, and leave forwarding type aliases in the sql package for compatibility. Similarly, if many packages want to loosely couple a common set of tags, you could imagine defining an interface to represent the tag, and having the independent tag structs implement that interface. This opens up all of our usual tools in API design, and makes tags amenable to them. That seems to me to be a huge advantage of this proposal, and one that would be hard to replicate without inventing lots of concepts that would be very similar to the familiar notion of structs/interfaces/type-aliases/etc.

[1] For the record, I really don't think this sort of sharing is a good idea (I'm fairly sure one of the very first entries on the protobuf API best practices list is "don't use the same protos for storage and for clients", or something to that effect; it introduces a lot of coupling between systems that evolve very differently (compare how often a backend is deployed (maybe daily) with how often a client might update (maybe never, for users who don't update their apps on their phone or if the WASM is cached by the browser)), and does not allow you to evolve your storage (e.g. denormalize data) without making the difference visible to clients.

Contributor

balasanjay commented Jul 12, 2018

@leafbebop That counter-proposal seems to lack a significant amount of detail that makes it hard to evaluate; you should consider opening a separate proposal, if you think that is the way to go. Among other things, I don't understand how such "error checking" could be implemented, nor do I understand how packages would read that data (e.g. how would the json package read whether a field is tagged with omit empty). If the binding here are just strings, then how would we globally agree on who gets the "sql" string, or the "json" string, when validating a struct with a particular tag? There are multiple packages that deal with these subjects, and they may well want different data structures. Its also unclear how to version changes to these structures.

To discuss your objections:

  1. I agree that field tags are descriptive pieces of info, oriented to an "outer" program. But I don't understand how it follows that its philosophically wrong to bind the information to a concrete use. I also see little difference whether you say "json" as a string, or json as a package reference (which you import as a string "encoding/json"). In either case, you're clearly describing the exact same thing semantically. (And here, there is no init function, nor any loss in dead-code-elimination).

  2. Versioning and type aliases were presumably referenced to discuss how a change like this (or any approach which namespaces symbols) enables evolution in the ecosystem, using the same tools that we use for evolving structs. And until you address how your counterproposal would deal with that problem, it is hard to evaluate its effectiveness.

  3. Let's say that we grant that this use-case is a valid one[1]; to me, it seems that its a straightforward anti-pattern to define tag-structs in packages that have init functions (or call cgo, or use unsafe, or include a giant dependency tree). First, the language does not need to prevent every anti-pattern, it can't. Second, consider if you're the author of such a sql package and you get a bug report to this effect ("I'd like to use the tag without pulling in the whole DB driver"); it is rather straightforward to fix in a completely backwards-compatible manner. Simply add a new "sqltag" package (or whatever you want to call it), move the tag types in there, and leave forwarding type aliases in the sql package for compatibility. Similarly, if many packages want to loosely couple a common set of tags, you could imagine defining an interface to represent the tag, and having the independent tag structs implement that interface. This opens up all of our usual tools in API design, and makes tags amenable to them. That seems to me to be a huge advantage of this proposal, and one that would be hard to replicate without inventing lots of concepts that would be very similar to the familiar notion of structs/interfaces/type-aliases/etc.

[1] For the record, I really don't think this sort of sharing is a good idea (I'm fairly sure one of the very first entries on the protobuf API best practices list is "don't use the same protos for storage and for clients", or something to that effect; it introduces a lot of coupling between systems that evolve very differently (compare how often a backend is deployed (maybe daily) with how often a client might update (maybe never, for users who don't update their apps on their phone or if the WASM is cached by the browser)), and does not allow you to evolve your storage (e.g. denormalize data) without making the difference visible to clients.

@urandom

This comment has been minimized.

Show comment
Hide comment
@urandom

urandom Jul 12, 2018

@leafbebop

The more I think about it, the more I don't see importing packages for tags as a problem.
Everything else staying the same, if a package with side effects has tags, they can be defined in a separate package so as not to cause problems when used in the model definitions. Since this is a new concept, tagged for Go2, potential rewrites to accommodate this shouldn't be taken negatively into account.

And even if they don't, perhaps the compiler will be able to remove the coupling when compiling your modules. It will know that the struct tags are only used during runtime, and if it sees that these structs are not passed to any other piece of code from the third party package, it could probably deduce that the import is only for tags and essentially remove it, thus not executing its init scripts at all.

Finally, considering the equivalent use cases of annotations in other languages (Java, C# come to mind from the ones I've used), I've not seen problems being raised by having such coupling. There, the annotations are defined structures that you have to import them from their defining packages.

As for code reuse, I've only read about that in articles that deal with nodejs for now. So far I haven't seen the same codebase being shared in any other environment. Of course, that's all anecdotal.

EDIT: As for the yaml problem. I don't really see that as a problem as well. FIrst, there are no guarantees that all the yaml parsers right now use the same tag, let alone the same format even if the name was the same. Second, once you settle on a yaml parser, its unlikely that you will switch to a different one later down the road. Same with databases, once you pick one, you usually stick with it. I've personally yet to see a client want to switch to a different database down the road (anecdotal).

urandom commented Jul 12, 2018

@leafbebop

The more I think about it, the more I don't see importing packages for tags as a problem.
Everything else staying the same, if a package with side effects has tags, they can be defined in a separate package so as not to cause problems when used in the model definitions. Since this is a new concept, tagged for Go2, potential rewrites to accommodate this shouldn't be taken negatively into account.

And even if they don't, perhaps the compiler will be able to remove the coupling when compiling your modules. It will know that the struct tags are only used during runtime, and if it sees that these structs are not passed to any other piece of code from the third party package, it could probably deduce that the import is only for tags and essentially remove it, thus not executing its init scripts at all.

Finally, considering the equivalent use cases of annotations in other languages (Java, C# come to mind from the ones I've used), I've not seen problems being raised by having such coupling. There, the annotations are defined structures that you have to import them from their defining packages.

As for code reuse, I've only read about that in articles that deal with nodejs for now. So far I haven't seen the same codebase being shared in any other environment. Of course, that's all anecdotal.

EDIT: As for the yaml problem. I don't really see that as a problem as well. FIrst, there are no guarantees that all the yaml parsers right now use the same tag, let alone the same format even if the name was the same. Second, once you settle on a yaml parser, its unlikely that you will switch to a different one later down the road. Same with databases, once you pick one, you usually stick with it. I've personally yet to see a client want to switch to a different database down the road (anecdotal).

@leafbebop

This comment has been minimized.

Show comment
Hide comment
@leafbebop

leafbebop Jul 12, 2018

I will try one last time to repeat myself.

There are other problems about binding tags to packages and I have stated them before, but the most essential problem is, it is against the Go philosophy and many good programme principals.

First is the Single Responsibility Principle. Data model package, does not has the responsibilty to validate a certain outter logic schema, and thus shall not import that package.

Next is Open / Closed Principle. It reads "Software entities should be open for extension, but closed for modification".. Now as long as there is no way to modify or add field tags, the close part is fulfilled. However, binding the tag system to a concrete package denies open to extension, not to the data model itself but to packages that use the tags. All other package that needs to proceed a certain tag must import a certain concrete package that is mostly of the same usage of itself is not idiomatic, and the fact that it can only process the tag in the same way that certain package is designed? It is very hindering. ( Say I would like write a json parsing package which use omitempty as a string has three values "nil, empty, false". It is not a good design, but a viable usage, which will be hindered because encoding/json asks json:omitempty to be a bool - this is just to explain my idea, not a real world example) .

Then is Liskov Substitution Principle. Two things (I go a step further, it was orignally about type or class) are substitutable if they exhibit behaviour such that the caller is unable to tell the difference. So if you binds tags json to encoding/json, then in other package, where you auctually marshal the data, package encoding/json is not substituable without requiring code change.

Follows the Interface Segregation Principle. It reads "Clients should not be forced to depend on methods they do not use". It is almost obvious. Data model codes should not be forced to depend on models they do not use.

Here come the last, Dependency Inversion Principle. It reads "High-level modules should not depend on low-level modules. Both should depend on abstractions." Here, I would like to say, data models should not depend on low-level modules, be it sqlx or yaml, but rather, through an abstraction, which in this case can be a string, either a keyword style or the orignal style.

Now I have covered all SOLID principles, I think it is also worthwhile to mention the Go Proverb: "A little copying is better than a little dependency". Rob Pike himself said that keeping the dependency tree smaller can mean the program is able to "compile faster, easier to maintain, and simpler". It might be better these days thanks to all tools we have at hand, but the Proverb still is a thing.

As for my "counter-proposal", I am neglecting details on purpose. I myself is in doubt of whether is good - it is far more too verbose for something meaning to be a littile addon descriptive information of a certain field. On further reflection, I think either keeps it a string literal but adds certain restriction on it or solve the need of that piece of information elsewhere is better than a complicate tag system. Though I am uncertain about these, it feels real wrong to me to bind tag system to package.

leafbebop commented Jul 12, 2018

I will try one last time to repeat myself.

There are other problems about binding tags to packages and I have stated them before, but the most essential problem is, it is against the Go philosophy and many good programme principals.

First is the Single Responsibility Principle. Data model package, does not has the responsibilty to validate a certain outter logic schema, and thus shall not import that package.

Next is Open / Closed Principle. It reads "Software entities should be open for extension, but closed for modification".. Now as long as there is no way to modify or add field tags, the close part is fulfilled. However, binding the tag system to a concrete package denies open to extension, not to the data model itself but to packages that use the tags. All other package that needs to proceed a certain tag must import a certain concrete package that is mostly of the same usage of itself is not idiomatic, and the fact that it can only process the tag in the same way that certain package is designed? It is very hindering. ( Say I would like write a json parsing package which use omitempty as a string has three values "nil, empty, false". It is not a good design, but a viable usage, which will be hindered because encoding/json asks json:omitempty to be a bool - this is just to explain my idea, not a real world example) .

Then is Liskov Substitution Principle. Two things (I go a step further, it was orignally about type or class) are substitutable if they exhibit behaviour such that the caller is unable to tell the difference. So if you binds tags json to encoding/json, then in other package, where you auctually marshal the data, package encoding/json is not substituable without requiring code change.

Follows the Interface Segregation Principle. It reads "Clients should not be forced to depend on methods they do not use". It is almost obvious. Data model codes should not be forced to depend on models they do not use.

Here come the last, Dependency Inversion Principle. It reads "High-level modules should not depend on low-level modules. Both should depend on abstractions." Here, I would like to say, data models should not depend on low-level modules, be it sqlx or yaml, but rather, through an abstraction, which in this case can be a string, either a keyword style or the orignal style.

Now I have covered all SOLID principles, I think it is also worthwhile to mention the Go Proverb: "A little copying is better than a little dependency". Rob Pike himself said that keeping the dependency tree smaller can mean the program is able to "compile faster, easier to maintain, and simpler". It might be better these days thanks to all tools we have at hand, but the Proverb still is a thing.

As for my "counter-proposal", I am neglecting details on purpose. I myself is in doubt of whether is good - it is far more too verbose for something meaning to be a littile addon descriptive information of a certain field. On further reflection, I think either keeps it a string literal but adds certain restriction on it or solve the need of that piece of information elsewhere is better than a complicate tag system. Though I am uncertain about these, it feels real wrong to me to bind tag system to package.

@balasanjay

This comment has been minimized.

Show comment
Hide comment
@balasanjay

balasanjay Jul 12, 2018

Contributor

@leafbebop Alright, I think we're going to have to agree to disagree.

Most of these seem either misapplied (e.g. LSP isn't saying that any two arbitrary software constructs should be substitutable, just types and subtypes; where for packages, there's no such thing as subpackages, and it just generally seems to be talking about an entirely different domain) or contradictory (e.g. this definition of LSP contradicts the desire stated in OCP where some package invents their own notion of json's omitempty tag) or are "issues" with the current state of the world (e.g. SRP is equally violated if your data model has any functionality and has json string tags on it).

And there are more fundamental problems with the current state of the world than dreamed off by the authors of these principles; for instance, the compiler cannot help you if you typed "jsn" (after all, some "outer" package might legitimately be interpreting these), or if you include commas instead of spaces (see bug linked above).

I'm happy to admit the Proverb is relevant; it is legitimately a downside that there are potentially more dependencies being taken (though, again, keep in mind that this could be entirely mitigated if authors feel strongly via isolating tag structs in their own packages). But compared to the upside, this downside feels incredibly minor (to the point of insignificance).

Contributor

balasanjay commented Jul 12, 2018

@leafbebop Alright, I think we're going to have to agree to disagree.

Most of these seem either misapplied (e.g. LSP isn't saying that any two arbitrary software constructs should be substitutable, just types and subtypes; where for packages, there's no such thing as subpackages, and it just generally seems to be talking about an entirely different domain) or contradictory (e.g. this definition of LSP contradicts the desire stated in OCP where some package invents their own notion of json's omitempty tag) or are "issues" with the current state of the world (e.g. SRP is equally violated if your data model has any functionality and has json string tags on it).

And there are more fundamental problems with the current state of the world than dreamed off by the authors of these principles; for instance, the compiler cannot help you if you typed "jsn" (after all, some "outer" package might legitimately be interpreting these), or if you include commas instead of spaces (see bug linked above).

I'm happy to admit the Proverb is relevant; it is legitimately a downside that there are potentially more dependencies being taken (though, again, keep in mind that this could be entirely mitigated if authors feel strongly via isolating tag structs in their own packages). But compared to the upside, this downside feels incredibly minor (to the point of insignificance).

@creker

This comment has been minimized.

Show comment
Hide comment
@creker

creker Jul 12, 2018

@leafbebop

which will be hindered because encoding/json asks json:omitempt

I don't see how. The proposal follows the usual Go syntax for referencing package identifiers. If you have a conflict you can use the same tools that Go provides now.

Data model codes should not be forced to depend on models they do not use.

depend on low-level modules, be it sqlx or yaml

As was mentioned, in the context of this proposal you would define tags in a separate package.

creker commented Jul 12, 2018

@leafbebop

which will be hindered because encoding/json asks json:omitempt

I don't see how. The proposal follows the usual Go syntax for referencing package identifiers. If you have a conflict you can use the same tools that Go provides now.

Data model codes should not be forced to depend on models they do not use.

depend on low-level modules, be it sqlx or yaml

As was mentioned, in the context of this proposal you would define tags in a separate package.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment