Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: net: add MarshalText/UnmarshalText to HardwareAddr #29678

Open
maja42 opened this issue Jan 11, 2019 · 22 comments · May be fixed by #34452

Comments

@maja42
Copy link

commented Jan 11, 2019

What version of Go are you using (go version)?

$ go version
go1.10 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What did you do?

I'm trying to unmarshal json content into net.IP and net.HardwareAddr.
Both types are actually of type []byte
Since net.IP implements json.Marshaler, it works as expected. However, net.HardwareAddr does not and fails with the error "illegal base64 data at input byte 2" (that's the position of the separator).

Here's an example:
https://play.golang.org/p/HOBBAyvpfrK

I found the google group discussion about adding the Marshaller to the net.IP type: https://groups.google.com/forum/#!topic/golang-nuts/io8aHJarm6U

What did you expect to see?

I expect net.HardwareAddr to be consistent with net.IP and implement json.Marshaler.
Strings in the form of "ab:cd:ef:ab:cd:ef" inside json should be parsable to MAC addresses.

@agnivade agnivade changed the title net, json: Add MarshalText/UnmarshalText to net.HardwareAddr proposal: net: Add MarshalText/UnmarshalText to HardwareAddr Jan 11, 2019
@gopherbot gopherbot added this to the Proposal milestone Jan 11, 2019
@agnivade agnivade changed the title proposal: net: Add MarshalText/UnmarshalText to HardwareAddr proposal: net: add MarshalText/UnmarshalText to HardwareAddr Jan 11, 2019
@gopherbot gopherbot added the Proposal label Jan 11, 2019
@agnivade

This comment has been minimized.

Copy link
Member

commented Jan 11, 2019

Considering there is a ParseMAC corresponding to ParseIP which is basically what the Unmarshaller for IP does, this seems like a reasonable request.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Jan 23, 2019

There are backwards compatibility concerns with adding MarshalText/UnmarshalText where they did not exist before. Was it possible to marshal these to JSON/Gob/XML before and then decode them back? If so, we can't break the old encoding. Or even if it was only possible to marshal, if that was a form other languages/tools knew, we can't change that either.

While we're on the topic, do we need to think about net.IPMask too?

@rsc rsc added the WaitingForInfo label Feb 6, 2019
@rsc

This comment has been minimized.

Copy link
Contributor

commented Feb 6, 2019

Waiting for someone to dig into whether there are compatibility issues here.

@agnivade

This comment has been minimized.

Copy link
Member

commented Feb 21, 2019

/cc @mikioh

@gopherbot

This comment has been minimized.

Copy link

commented Mar 6, 2019

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@gopherbot gopherbot closed this Mar 6, 2019
@mikioh mikioh removed the WaitingForInfo label Mar 7, 2019
@mikioh

This comment has been minimized.

Copy link
Contributor

commented Mar 7, 2019

Just FYI.

As far as I can see/remember;

  • textual representation for link- or below-layer identifiers varies,
  • the existing net.ParseMAC accepts only a few well-known representations, IEEE and some vendors, for the sake of convenience,
  • the existing net.HardwareAddr.String method returns the form described in IEEE Std 802-2001, Section 3.1.8 hexadecimal representation with the pick of "colons" instead of "hyphens",
  • though, MAC-48 is now obsolete. IEEE RA states that "... octets separated by hyphens; the IEEE RA refers to this as the hexadecimal (hex) representation." in https://standards.ieee.org/content/dam/ieee-standards/standards/web/documents/tutorials/eui.pdf
@mikioh mikioh reopened this Mar 7, 2019
@mikioh

This comment has been minimized.

Copy link
Contributor

commented Mar 7, 2019

While we're on the topic, do we need to think about net.IPMask too?

Also net.IPNet? I don't think so. As shown in #30264, there is a need for flexible IP address printer but simple, single-form text marshaler and unmarshaler might not fit for the actual needs, not only for classical telco sectors but tech sectors.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Mar 20, 2019

Still looking for comments on compatibility issues.

@klnusbaum

This comment has been minimized.

Copy link

commented Sep 21, 2019

@rsc I've dug through the git history of the go repo and I can't seem to find anything that would indicate any backwards compatibility issues. If there are specific things you'd like me to look for, I'm more than happy to, but at this point, I don't see any harm in adding this function.

klnusbaum added a commit to klnusbaum/go that referenced this issue Sep 21, 2019
The HardwareAddr type has the ability to be transformed to and from a
string. However, this capability is not exposed in a manner suitable
for use with various forms of marshalling and unmarshaling of text
(e.g. JSON or YAML). Let's add the proper functions so that
HardwareAddr implements the TextMarshaller and TextUnmarshaler
interfaces from the encoding package.

Fixes golang#29678
klnusbaum added a commit to klnusbaum/go that referenced this issue Sep 21, 2019
The HardwareAddr type has the ability to be transformed to and from a
string. However, this capability is not exposed in a manner suitable
for use with various forms of marshalling and unmarshaling of text
(e.g. JSON or YAML). Let's add the proper functions so that
HardwareAddr implements the TextMarshaller and TextUnmarshaler
interfaces from the encoding package.

Fixes golang#29678
@gopherbot

This comment has been minimized.

Copy link

commented Sep 21, 2019

Change https://golang.org/cl/196817 mentions this issue: net: add text marshalling and unmarshalling for HardwareAddr

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Sep 22, 2019

@klnusbaum The backward compatibility issue is that someone could have marshaled net.HardwareAddr values to JSON using Go 1.13, and might try to unmarshal them using Go 1.14. The unmarshaling would fail because the format would be different.

Is there some reason that that is impossible?

@klnusbaum

This comment has been minimized.

Copy link

commented Sep 22, 2019

@ianlancetaylor I'm a little confused here. If someone marshaled a hardware address on their own into JSON, why would Go be on the hook for being able to unmarshal that value with this newly introduced code? That seems somewhat unreasonable to me.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Sep 22, 2019

In general, it seems reasonable that if you marshal a net.HardwareAddr into JSON with one Go release, you should be unmarshal that JSON into a net.HardwareAddr in a subsequent release. As far as I know that works today--you can marshal and unmarshal a net.HardwareAddr.

@klnusbaum

This comment has been minimized.

Copy link

commented Sep 22, 2019

As far as I know that works today--you can marshal and unmarshal a net.HardwareAddr.

Ah, ok. So this I think is the source of my confusion. I'm unaware of any way to marshal or unmarshal a net.HardwareAddr to JSON (for example) using the existing Go or any other previous version of Go. Can you point me to where this is possible?

@klnusbaum

This comment has been minimized.

Copy link

commented Sep 22, 2019

unless you're talking about a developer manually calling String and then writing that out to JSON. And then later, getting the string back and manually calling ParseMAC on it. In which case, the solution I've proposed in my PR would indeed be backwards compatible.

@zigazeljko

This comment has been minimized.

Copy link

commented Sep 22, 2019

Can you point me to where this is possible?

net.HardwareAddr has a underlying type of []byte and is therefore marshaled/unmarshaled as a base64 string (default behavior of encoding/json for byte slices).

Since that is an unusual format for MAC addresses, I guess it wasn't widely used, so there shouldn't be many compatibility issues.

In general, it seems reasonable that if you marshal a net.HardwareAddr into JSON with one Go release, you should be unmarshal that JSON into a net.HardwareAddr in a subsequent release.

If that's an issue, the UnmarshalText method can use base64.Decode as a fallback in case net.ParseMAC fails.

@klnusbaum

This comment has been minimized.

Copy link

commented Sep 22, 2019

Ah, so in other words, we're saying that we must remain backwards compatible with the raw marshaling of bytes? i.e. the behavior of this (which will marshal using the default marshal behavior of []byte) should continue to work?

	addr, _ := net.ParseMAC("aa:bb:cc:dd:ee:ff")

	marshaledBytes, _ := json.Marshal(addr)
	
	var unmarshaled net.HardwareAddr
	json.Unmarshal(marshaledBytes, &unmarshaled)
	
	fmt.Printf("Unmarshalled Addr: %s", unmarshaled)

If that's the case, then yea, I think we can do as @zigazeljko suggested, fallback to raw byte unmarshaling in the case of ParseMAC failing. @ianlancetaylor do you think that would work?

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Sep 23, 2019

It's worth a try. Make sure to include a test with a net.HardwareAddr pre-encoded as if by an earlier release.

@klnusbaum

This comment has been minimized.

Copy link

commented Sep 24, 2019

@ianlancetaylor @zigazeljko I've added the ability to unmarshal the mac address in the case that it was originally encoded as a raw byte slice. I've also added a test to ensure such functionality actually works. I believe this should assuage any backwards compatibility fears. IMHO, it feels pretty silly and doesn't actually result in a better product. But if that is the will of the Go community so be it.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Sep 24, 2019

Why is backward compatibility silly?

@klnusbaum

This comment has been minimized.

Copy link

commented Sep 25, 2019

Oh, I'm not saying that backwards compatibility is silly over all. It's great overall. It just always comes with a cost. In my own projects, I would say the extra code here adds confusion and it's not worth supporting what I perceive to be an edge case from previous versions. That said, this seems to be the kind of tradeoff where, at least for the go project, the benefits outweigh the costs and so we do want to add the extra code.

klnusbaum added a commit to klnusbaum/go that referenced this issue Sep 30, 2019
The HardwareAddr type has the ability to be transformed to and from a
string. However, this capability is not exposed in a manner suitable
for use with various forms of marshalling and unmarshaling of text
(e.g. JSON or YAML). Let's add the proper functions so that
HardwareAddr implements the TextMarshaller and TextUnmarshaler
interfaces from the encoding package.

Fixes golang#29678
@klnusbaum

This comment has been minimized.

Copy link

commented Oct 9, 2019

Hey Folks,
I think the PR is in pretty good shape at the moment. Is there anything left for me to do? If not that's fine. Just want to make sure there aren't any further actions I need to take on my end at the moment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.