Feature request: json.RawMessage equivalent

[michaelsauter]

It would be great to have sth. similar to http://golang.org/pkg/encoding/json/#RawMessage.

I'm using go-yaml in http://github.com/michaelsauter/crane and would like to unmarshal the JSON/YAML partially (to provide defaults for objects inside arrays).

[GeorgeMac]

I would like to +1 this. Would be very useful for writing config files in my projects!

[fgrosse]

Yeah I am missing that too so I can delay the actual decoding of some fields.
Is there a known work around? I don't get how the Unmarshaller interface1 is supposed to work.

[ghost]

+1 for yaml.RawMessage

edit

The yaml.MapSlice structure seems to be a much better solution than json.RawMessage

[missedone]

+1 for yaml.RawMessage

[niemeyer]

The problem with yaml.RawMessage is that unlike JSON, YAML is context-dependent. One cannot properly encode a piece of raw YAML without knowing what the surrounding data is, due to indentation.

Can the +1 voters please detail a bit how this is intended to be used that isn't solved by the current system? Maybe we can design something else that solves the problem.

[ghost]

@niemeyer The problem I had intended to solve using RawMessage I had later solved using MapSlice. I am relatively new to golang, so this wasn't immediately obvious to me.

I think most problems people intend to solve with a json.RawMessage equivalent may be solved with yaml.MapSlice, but there isn't much documentation or example usages of it. If there was perhaps more of this, people will find that they do not need a yaml.RawMessage.

[missedone]

@niemeyer In my case, I have a lot of yaml configuration files, each includes some common definition and plugin settings, for plugin settings it's variant from plugin to plugin, i want to delay the parse and pass the RawMessage to leave the plugin implementation do it.
i know i can extract the plugin settings into separate yaml file, but it's really annoying i need to reference them in the parent yaml file.

[fgrosse]

I have the exact same use case as @missedone

[niemeyer]

Yeah, it should be easy to use a MapSlice, and one can even marshal it back into yaml and then unmarshal it again to get the exact semantics of a RawMessage. That said, I'll keep this issue open so we can consider the idea of a RawMessage-like feature further.

[niemeyer]

Thanks for all the feedback, by the way!

[dvirsky]

It would be great if you could just have a method mapping a MapSlice (or a plain map[string]interface{} for that matter) directly into a struct, without the need to re-marshal it.

[niemeyer]

@dvirsky That goes beyond the goals of the yaml package, but there are other packages available for such direct map-to-struct conversion (e.g. https://github.com/mitchellh/mapstructure).

[tchap]

Isn't it enough to implement UnmarshalYAML and store the unmarshal function for later use and just immediately return from the function? I tried this and it works to just call the function later.

type RawMessage struct {
    unmarshal func(interface{}) error
}

func (msg *RawMessage) UnmarshalYAML(unmarshal func(interface{}) error) error {
    msg.unmarshal = unmarshal
    return nil
}

func (msg *RawMessage) Unmarshal(v interace{}) error {
    return msg.unmarshal(v)
}

Then you can call RawMessage.Unmarshal(v) later and perform the actual unmarshalling, right?

[niemeyer]

Yes, that's similar to what bson.Raw does, but it's not quite the same, and it's not the same as json.RawMessage either. The problem is that unlike bson and json, you cannot marshal it back as-is, because yaml marshaling is context sensitive.

It's likely the direction we'll take, though. We'll just need some further work on the marshaling side.

[tchap]

True, I only needed to defer unmarshalling, was not really thinking about marshalling, but sure, it's obviously not symmetrical.

[niemeyer]

And not very "Raw", if you see what I mean. We'll likely end up with a similar abstraction that can lazily unmarshal and re-marshal appropriately, but not provide direct access to the text.

[tchap]

Yep.

[andrewrynhard]

Are there any good examples of how to use yaml.MapSlice?

[perj]

What I wanted to use json.RawMessage for was to have a json dictionary in the middle of the YAML output. That's still valid yaml. E.g. to output this:

foo:
  bar: { "a" : 1 }
  baz:
    b: 2

Using only map[string]interface{} types the code would look like this (untested):

topdata := make(map[string]interface{})
foodata := make(map[string]interface{})
topdata["foo"] = foodata
foodata["bar"] = json.RawMessage(`{ "a" : 1 }`)
foodata["baz"] = map[string]interface{}{ "b" : 2 }
enc, _ := yaml.Marshal(topdata)
println(enc)

Instead, the json.RawMessage seems to be interpreted as a slice of bytes and each byte output as an integer element.

This package should at least support some way of doing the output above. As far as I can tell it isn't possible right now.

[rogpeppe]

I have some ideas for how this might work. It's possible that the YAML equivalent of json.RawMessage might be a parsed sequence of YAML "events" (e.g map start, scalar, sequence end, etc) which can then potentially be marshaled and unmarshaled in a context-insensitive way.

[excavador]

+1
I need it

[pete911]

not sure if this is going to help, but we started doing this in our case:

var configData = `
name: test
version: 0
plugins:
  http:
    user: joe
    url: https://test.com
  region:
    name: oz
`

type Config struct {
	Name    string `yaml:"name"`
	Version string `yaml:"version"`
	Plugins map[string]interface{} `yaml:"plugins"`
}

// custom plugins can be defined outside of this package
type HttpPlugin struct {
	User string `yaml:"user"`
	Url  string `yaml:"url"`
}

type RegionPlugin struct {
	Name string `yaml:"name"`
}

func TestABC(t *testing.T) {

	var config Config
	err := yaml.Unmarshal([]byte(configData), &config)
	require.NoError(t, err)
	fmt.Printf("unmarshaled config:\n%+v\n\n", config)

	// again custom plugin can be defined outside of this package
	var http HttpPlugin
	err = UnmarshalPlugin(config.Plugins["http"], &http)
	require.NoError(t, err)
	fmt.Printf("unmarshaled http plugin:\n%+v\n", http)

	var region RegionPlugin
	err = UnmarshalPlugin(config.Plugins["region"], &region)
	require.NoError(t, err)
	fmt.Printf("unmarshal region plugin:\n%+v\n", region)
}

// takes interface, marshals back to []byte, then unmarshals to desired struct
func UnmarshalPlugin(pluginIn, pluginOut interface{}) error {

	b, err := yaml.Marshal(pluginIn)
	if err != nil {
		return err
	}
	return yaml.Unmarshal(b, pluginOut)
}

basically use map[string]interface{} instead of raw message, and have UnmarshalPlugin

[clafoutis42]

@pete911 this is a good idea, this might work for me, thanks !

However, a type json.RawMessage like (which would most likely be named yaml.RawMessage) would be welcomed.

[luzhzhsoft]

+1
i need it.

[aloababa]

+1

[xemul]

I've just tried the method proposed by @tchap and it worked perfectly! If this could be included into the official yaml package, then the need for the RawMessage just goes away, at least from my side :)

[rssllyn]

You can define a string field for such a RawMessage. Then, in your yaml file, write a multi-line string as value of that field.

reference: https://stackoverflow.com/a/21699210/1033539

[ake-persson]

+1

[quenbyako]

@niemeyer Any status of this feature?

[niemeyer]

We won't have a RawMessage because unlike json yaml is context and indentation sensitive, so we can't just write out some string as-is without pre-processing. That said, with v3 we have yaml.Node, which offers something convenient for preserving more of the structure of some document.

[quenbyako]

@niemeyer thanks for responding! Not so cool to hear that, but i hope third version of this package will help a lot with dynamic typed configs.

Also thanks a lot for this lib! You're doing amazing job 😃

[davidmdm]

I came upon this problem today, and it is easily solvable. I solved it for myself within the consuming package but it could be a nice addition to the main repository. However I know that PRs are not generally accepted or reviewed so maybe I will add it to my fork, but for all those who want a solution, simply we want to use the UnmarshalYAML(node *yaml.Node) error interface and simply delay parsing of that node.

Here is a full working example:

package main

import (
	"fmt"

	"gopkg.in/yaml.v3"
)

func main() {
	data := `key: raw message equivalent complete!`

	var target map[string]RawNode
	if err := yaml.Unmarshal([]byte(data), &target); err != nil {
		panic(err)
	}

	var str string
	if err := target["key"].Decode(&str); err != nil {
		panic(err)
	}

	fmt.Println(str) // Print: raw message equivalent complete!
}

type RawNode struct{ *yaml.Node }

func (n *RawNode) UnmarshalYAML(node *yaml.Node) error {
	n.Node = node
	return nil
}