Skip to content

Commit

Permalink
Support encoding comments and specifying the encoding format
Browse files Browse the repository at this point in the history
This allows encoding comments and setting some flags to control the
format. While toml.Marshaler added in #327 gives full control over how
you want to format something, I don't think it's especially
user-friendly to tell everyone to create a new type with all the
appropriate formatting, escaping, etc. The vast majority of of use cases
probably just call in to a few simple categories such as "use `"""` or
"encode this number as a hex".

I grouped both features together as they're closely related: both set
additional information on how to write keys.

What I want is something that:

1. allows setting attributes programmatically;
2. supports round-tripping by default on a standard struct;
3. plays well with other encoders/decoders;
4. has a reasonable uncumbersome API.

Most options (custom types, struct tags) fail at least one of these;
there were some PRs for struct tags, but they fail at 1, 2, and
(arguably) 4. Custom types fail at 2, 3, and probably 4.

---

This adds SetMeta() to the Encoder type; this is already what we have
when decoding, and all additional information will be set on it. On
MetaData we add the following types:

    SetType()       Set TOML type info.
    TypeInfo()      Get TOML type info.
    Doc()           Set "doc comment" above the key.
    Comment()       Set "inline comment" after the key.

Every TOML type has a type in this package, which support different
formatting options (see type_toml.go):

    Bool
    String
    Int
    Float
    Datetime
    Table
    Array
    ArrayTable

For example:

    meta := toml.NewMetaData().
            SetType("key", toml.Int{Width: 4, Base: 16}).
            Doc("key", "A codepoint").
    Comment("key", "ë")
    toml.NewEncoder(os.Stdout).SetMeta(meta).Encode(struct {
            Key string `toml:"key"`
    }{"ë")

Would write:

    # A codepoint.
    key = 0x00eb  # ë

It also has Key() to set both:

    toml.NewMetaData().
        Key("key", toml.Int{Width: 4, Base: 16}, toml.Doc("A codepoint"), toml.Comment("ë")).
        Key("other", toml.Comment("..."))

The advantage of this is that it reduces the number of times you have to
type the key string to 1, but it uses interface{}. Not yet decided which
one I'll stick with, and also not a huge fan of Doc() and Comment(), but
I can't really think of anything clearer at the moment (these are the
names the Go ast uses).

---

The Decode() sets all this information on the MetaData, so this:

    meta, _ := toml.Decode(..)
    toml.NewEncoder(os.Stdout).SetMeta(meta).Encode(..)

Will write it out as "key = 0x00eb" again, rather than "key = 235".

This way, pretty much any flag can be added programmatically without
getting in the way of JSON/YAML/whatnot encoding/decoding.

---

I don't especially care how you need to pass the keys as strings, but
there isn't really any good way to do it otherwise. There is also the
problem that the "key" as found in the parser may be different than the
"key" the user is expecting if you don't use toml struct tags:

    type X struct { Key int }

Will read "key = 2" in to "Key", but when encoding it will write as
"Key" rather than "key". The type information will be set to "key", but
when encoding it will look for "Key", so round-tripping won't work
correct and has the potential for confusion if the wrong key is set.

This is not so easy to fix since we don't have access to the struct in
the parser. I think it's fine to just document this as a caveat and tell
people to use struct tags, which is a good idea in any case.

---

I'm not necessarily opposed to also adding struct tags for most of these
things, although I'm not a huge fan of them. Since struct tags can't be
set programmatically it's not really suitable for many use cases (e.g.
setting comments dynamically, using multiline strings only if the string
contains newlines, etc.) It's something that could maybe be added in a
future PR, if a lot of people ask for it.

Fixes #64
Fixes #75
Fixes #160
Fixes #192
Fixes #213
Fixes #269
  • Loading branch information
arp242 committed Nov 24, 2021
1 parent 7eb955f commit 030ae5e
Show file tree
Hide file tree
Showing 14 changed files with 1,063 additions and 546 deletions.
5 changes: 3 additions & 2 deletions cmd/toml-test-decoder/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,13 +31,14 @@ func main() {
}

var decoded interface{}
if _, err := toml.DecodeReader(os.Stdin, &decoded); err != nil {
meta, err := toml.DecodeReader(os.Stdin, &decoded)
if err != nil {
log.Fatalf("Error decoding TOML: %s", err)
}

j := json.NewEncoder(os.Stdout)
j.SetIndent("", " ")
if err := j.Encode(tag.Add("", decoded)); err != nil {
if err := j.Encode(tag.Add(meta, "", decoded)); err != nil {
log.Fatalf("Error encoding JSON: %s", err)
}
}
39 changes: 14 additions & 25 deletions decode.go
Original file line number Diff line number Diff line change
Expand Up @@ -46,23 +46,6 @@ const (
maxSafeFloat64Int = 9007199254740991 // 2^53-1
)

// PrimitiveDecode is just like the other `Decode*` functions, except it
// decodes a TOML value that has already been parsed. Valid primitive values
// can *only* be obtained from values filled by the decoder functions,
// including this method. (i.e., `v` may contain more `Primitive`
// values.)
//
// Meta data for primitive values is included in the meta data returned by
// the `Decode*` functions with one exception: keys returned by the Undecoded
// method will only reflect keys that were decoded. Namely, any keys hidden
// behind a Primitive will be considered undecoded. Executing this method will
// update the undecoded keys in the meta data. (See the example.)
func (md *MetaData) PrimitiveDecode(primValue Primitive, v interface{}) error {
md.context = primValue.context
defer func() { md.context = nil }()
return md.unify(primValue.undecoded, rvalue(v))
}

// Decoder decodes TOML data.
//
// TOML tables correspond to Go structs or maps (dealer's choice – they can be
Expand Down Expand Up @@ -116,8 +99,8 @@ func (dec *Decoder) Decode(v interface{}) (MetaData, error) {
return MetaData{}, e("Decode of nil %s", reflect.TypeOf(v))
}

// TODO: have parser should read from io.Reader? Or at the very least, make
// it read from []byte rather than string
// TODO: parser should read from io.Reader? Or at the very least, make it
// read from []byte rather than string
data, err := ioutil.ReadAll(dec.r)
if err != nil {
return MetaData{}, err
Expand All @@ -128,8 +111,12 @@ func (dec *Decoder) Decode(v interface{}) (MetaData, error) {
return MetaData{}, err
}
md := MetaData{
p.mapping, p.types, p.ordered,
make(map[string]bool, len(p.ordered)), nil,
mapping: p.mapping,
types: p.types,
keys: p.ordered,
comments: p.comments,
decoded: make(map[string]bool, len(p.ordered)),
context: nil,
}
return md, md.unify(p.mapping, indirect(rv))
}
Expand Down Expand Up @@ -258,17 +245,17 @@ func (md *MetaData) unifyStruct(mapping interface{}, rv reflect.Value) error {
for _, i := range f.index {
subv = indirect(subv.Field(i))
}

if isUnifiable(subv) {
md.decoded[md.context.add(key).String()] = true
md.context = append(md.context, key)
if err := md.unify(datum, subv); err != nil {
err := md.unify(datum, subv)
if err != nil {
return err
}
md.context = md.context[0 : len(md.context)-1]
} else if f.name != "" {
// Bad user! No soup for you!
return e("cannot write unexported field %s.%s",
rv.Type().String(), f.name)
return e("cannot write unexported field %s.%s", rv.Type().String(), f.name)
}
}
}
Expand Down Expand Up @@ -459,6 +446,7 @@ func (md *MetaData) unifyText(data interface{}, v encoding.TextUnmarshaler) erro
var s string
switch sdata := data.(type) {
case Marshaler:
fmt.Println("unifyText (Marshaler)", data, "in to", v)
text, err := sdata.MarshalTOML()
if err != nil {
return err
Expand All @@ -470,6 +458,7 @@ func (md *MetaData) unifyText(data interface{}, v encoding.TextUnmarshaler) erro
return err
}
s = string(text)
// fmt.Println("unifyText (TextMarshaler)", data, "in to", v, "=", s)
case fmt.Stringer:
s = sdata.String()
case string:
Expand Down
123 changes: 0 additions & 123 deletions decode_meta.go

This file was deleted.

Loading

0 comments on commit 030ae5e

Please sign in to comment.