Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

encoding/xml: support for XML namespace prefixes #9519

Open
jordan2175 opened this issue Jan 6, 2015 · 24 comments
Open

encoding/xml: support for XML namespace prefixes #9519

jordan2175 opened this issue Jan 6, 2015 · 24 comments

Comments

@jordan2175
Copy link

@jordan2175 jordan2175 commented Jan 6, 2015

Marshal-ing data back to XML does not seem to support namespace prefixes.. For example:

https://play.golang.org/p/6CY71H7mb4

The input XML is: <stix:STIX_Package>, but when it writes it back out it does <STIX_Package xmlns="stix">

Also, there does not appear to be any data elements like xml.Name for adding namespaces to a struct.... Something maybe like xml.NS????

@mikioh mikioh changed the title Support for XML namespace prefixes encoding/xml: Support for XML namespace prefixes Jan 6, 2015
@mikioh

This comment has been minimized.

Copy link
Contributor

@mikioh mikioh commented Jan 6, 2015

Dup of #6800

@mikioh mikioh closed this Jan 6, 2015
@jordan2175

This comment has been minimized.

Copy link
Author

@jordan2175 jordan2175 commented Jan 6, 2015

I think what I am looking for, as I do more research on it, is for xml.Marshal to support prefixes

@jordan2175

This comment has been minimized.

Copy link
Author

@jordan2175 jordan2175 commented May 29, 2015

Here is some more details about this issue: If you run this code:

https://play.golang.org/p/EucDh59yiB

package main

import (
    "encoding/xml"
    "fmt"
)

type HouseType struct {
    XMLName   xml.Name `xml:"prefix11 House"`
    MessageId string   `xml:"message_id,attr"`
}

func main() {

    var tm HouseType
    tm.MessageId = "test1234"

    var data1 []byte
    data1, _ = xml.MarshalIndent(tm, "", "    ")

    fmt.Println("Marshal")
    fmt.Println(string(data1))

    rawxml := `<prefix11:House message_id="1466" in_response_to="1"></prefix11:House>`

    var tm2 HouseType
    xml.Unmarshal([]byte(rawxml), &tm2)
    fmt.Println("\nUnmarshal")
    fmt.Println("Message ID", tm2.MessageId)
}

It will print out:

Marshal
<House xmlns="prefix11" message_id="test1234"></House>

Unmarshal
Message ID 1466

You will see that the marshal command does not make use of the prefix correctly. It should be prefix11:House. If I change the struct to look like the following then the marshal command will work but the unmarshal will not. NOTE the ":" in the XMLName field. So I can either marshal or unmarshal, but not both with prefixed XML

https://play.golang.org/p/44CMHXb3YM

type HouseType struct {
    XMLName   xml.Name `xml:"prefix11:House"`
    MessageId string   `xml:"message_id,attr"`
}

Making this change, changes the output to:

Marshal
<prefix11:House message_id="test1234"></prefix11:House>

Unmarshal
Message ID
@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Jul 23, 2015

See #11841.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Jul 23, 2015

CL https://golang.org/cl/12570 mentions this issue.

rsc added a commit that referenced this issue Jul 27, 2015
There is clearly work to do here with respect to xml name spaces,
but I don't believe the changes in this cycle are clearly correct.
The changes in this cycle have visible impact on the generated xml,
possibly breaking existing programs, and yet it's not clear that they
are the end of the story: there is still significant confusion about how
name spaces work or should work (see #9519, #9775, #8167, #7113).

I would like to wait to make breaking changes until we completely
understand what the behavior should be and can evaluate the benefit
of those breaking changes. My main concern here is that we will break
programs in Go 1.5 for the sake of name space adjustments and then
while trying to fix those other bugs we'll break programs in Go 1.6 too.
Let's wait until we know all the changes we want to make before we
decide whether or how to break existing programs.

This CL reverts:

5ae822b encoding/xml: minor changes
bb7e665 encoding/xml: fix xmlns= behavior
9f9d66d encoding/xml: fix default namespace of tags
b69ea01 encoding/xml: fix namespaces in a>b tags
3be158d encoding/xml: encoding name spaces correctly

and adjusts tests from

a9dddb5 encoding/xml: add more EncodeToken tests.

to expect Go 1.4 behavior.

I have confirmed that the name space parts of the test suite
as of this CL passes against the Go 1.4 encoding/xml package,
indicating that this CL successfully restores the Go 1.4 behavior.

(Other tests do not, but that's because there were some real
bug fixes in this cycle that are being kept. Specifically, the
tests that don't pass in Go 1.4 are TestMarshal's NestedAndComment
case, TestEncodeToken's encoding of newlines, and
TestSimpleUseOfEncodeToken returning an error for invalid
token types.)

I also checked that the Go 1.4 tests pass when run against
this copy of the sources.

Fixes #11841.

Change-Id: I97de06761038b40388ef6e3a55547ff43edee7cb
Reviewed-on: https://go-review.googlesource.com/12570
Reviewed-by: Nigel Tao <nigeltao@golang.org>
@rsc rsc changed the title encoding/xml: Support for XML namespace prefixes encoding/xml: support for XML namespace prefixes Nov 5, 2015
@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Nov 25, 2015

Blocked on #13400.

@rothskeller

This comment has been minimized.

Copy link

@rothskeller rothskeller commented Jun 14, 2016

Just a note in support of this proposal: with encoding/xml as it stands, it is infeasible to interact with SOAP web services that use digitally signed requests. This is because XML digital signatures require the XML to be in canonical form, which requires a greater degree of control over namespaces than encoding/xml provides. Admittedly, calling a SOAP service from a Go program is kind of like towing a trailer behind your Ferrari, but sometimes interoperability with legacy systems is essential.

@vania-pooh

This comment has been minimized.

Copy link

@vania-pooh vania-pooh commented Oct 7, 2016

Just a workaround while waiting for a fix:

type Data ...

type Root struct {
    XMLName  xml.Name  `xml:"prefix:root"`
    XmlNS    string    `xml:"xmlns:prefix,attr"`
    Data Data `xml:"data"`
}

root := Root {
    XmlNS: "urn:test.example.com",
    Data: ...,
}

b, err := xml.MarshalIndent(root, "", "    ")

This will produce the following:

<prefix:root xmlns:prefix="urn:test.example.com"/>
    <data>...</data>
</prefix:root>
@karl-gustav

This comment has been minimized.

Copy link

@karl-gustav karl-gustav commented Jan 23, 2017

@vania-pooh Am I right in assuming that I can't unmarshal a namespaced tag into a namespaced struct? (I.e. go sees them as two different objects)

Here's an example of what I mean, here "urn:copyright" is not unmarshaled, but if the name is changed to "copyright" is: https://play.golang.org/p/lb1oZ0ATwz

@vania-pooh

This comment has been minimized.

Copy link

@vania-pooh vania-pooh commented Jan 23, 2017

@karl-gustav not sure, I played only with marshalling. But so far as I understand currently Go supports only prefixed attributes and does not support prefixed tag names.

@uynap

This comment has been minimized.

Copy link

@uynap uynap commented Mar 14, 2017

@vania-pooh It's apparently not a valid "workaround". Marshal and UnMarshal are not working with the same tag.
In your example, you have to use XMLName xml.Name xml:"prefix:root" for Marshal and XMLName xml.Name xml:"prefix root" for UnMarshal. (Go version 1.8)

@vania-pooh

This comment has been minimized.

Copy link

@vania-pooh vania-pooh commented Mar 14, 2017

@uynap didn't test with unmarshal. This was marshalling only workaround.

@karl-gustav

This comment has been minimized.

Copy link

@karl-gustav karl-gustav commented Mar 31, 2017

Do we have a minimal working example of a full Marshal / UnMarshal workaround ?

@uynap

This comment has been minimized.

Copy link

@uynap uynap commented Apr 1, 2017

@karl-gustav The issue for Golang "XML namespace prefixes" is that you cannot use one struct for both Marshal and UnMarshal. But it's fairly easy to have the code only supports Marshal or UnMarshal.
For example, when you do Marshal use the code below:

type XMLenvelop struct {
    XMLName xml.Name `xml:"soapenv:Envelope"`
}

for UnMarshal:

type XMLenvelop2 struct {
    XMLName xml.Name
}
dmlambea added a commit to dmlambea/gowsdl that referenced this issue May 16, 2017
…eader, WSSUsernameToken and its child structs WSSUsername and WSSPassword.

XML namespace problems are dealt with by using vania-pooh's workaround:  golang/go#9519 (comment)

Convenience function "NewWSSSecurityHeader" to ease header creation (the header is not automatically added to the SOAP client).

"SetHeader" method is kept for compatibility. There is a new "AddHeader" method to add headers to the SOAP call. There is now support for several headers instead of only one.

Cosmetic changes in header_tmpl.go to sort import names alphabetically.

Sample code:

   ...
   hdr := myservice.NewWSSSecurityHeader("sampleuser", "samplepassword", "1")  // "1" is for 'mustUnderstand' attribute; empty if not specified
   myserviceClient.AddHeader(hdr)
   ...
@Zauberstuhl

This comment has been minimized.

Copy link

@Zauberstuhl Zauberstuhl commented Jan 22, 2018

Hi,

since I really wanted this in my private project I started playing around and
I actually have a working version for me. Could someone with deeper knowledge of the lib take a look at the changes and tell me whether that actually makes sense (Zauberstuhl@eb84a70)?

Here is an example of how that could look like:
https://play.golang.org/p/O4Ene-8GrVV

Cheers

@lkzcover

This comment has been minimized.

Copy link

@lkzcover lkzcover commented Feb 12, 2018

@sophos this works for XML-tags, but do not work for XML-attributes

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Jun 4, 2018

Change https://golang.org/cl/116056 mentions this issue: encoding/xml: fix printing of namespace prefix in tag names

@iWdGo

This comment has been minimized.

Copy link
Contributor

@iWdGo iWdGo commented Jun 4, 2018

With the submitted fix, prefix displays when defined in tag names, i.e. no made up prefix is taken into account. The URL of the name space is also returned for an End Token as documentation requires.
Translating the prefix is popping the NS which is unavailable for translate. Translate for an End Token has been moved inside the pop element part.

When a tag.Space has no prefix, it is the default space xmlns=".Space" according to documentation. Since there is no prefix the print remains <tag.Name.local … and not <Space:.Local…

The fix works on top of the merged fixes of the list issues in #13400 as namespace standard needs to be enforced before improving handling.

@Nerzal

This comment has been minimized.

Copy link

@Nerzal Nerzal commented Oct 10, 2018

Just a workaround while waiting for a fix:

type Data ...

type Root struct {
    XMLName  xml.Name  `xml:"prefix:root"`
    XmlNS    string    `xml:"xmlns:prefix,attr"`
    Data Data `xml:"data"`
}

root := Root {
    XmlNS: "urn:test.example.com",
    Data: ...,
}

b, err := xml.MarshalIndent(root, "", "    ")

This will produce the following:

<prefix:root xmlns:prefix="urn:test.example.com"/>
    <data>...</data>
</prefix:root>

This workaround only works when marshaling, but fails when you want to unmarshal :/

@genez

This comment has been minimized.

Copy link

@genez genez commented Nov 8, 2018

I have the same issue, while interacting with an Italian government service.

This playground demonstrate the issue: https://play.golang.org/p/H5Hibbci81_n

@lelvisl

This comment has been minimized.

Copy link

@lelvisl lelvisl commented May 7, 2019

@nimish

This comment has been minimized.

Copy link

@nimish nimish commented Nov 30, 2019

For the record, the prefix shouldn't actually matter. If two prefixes resolve to the same namespace then they are identical, including the empty prefix.

https://www.w3.org/TR/xml-names/#NT-PrefixedName

Note that the prefix functions only as a placeholder for a namespace name. Applications SHOULD use the namespace name, not the prefix, in constructing names whose scope extends beyond the containing document.

The full solution here is for the package to support full qualified names, and then have some notion of a map between Namespaces and Prefixes that is determined at marshal-time.

<prefix:root xmlns:prefix="urn:test.example.com"/>
</prefix:root>

Is (or should be) treated identically to:

<root xmlns="urn:test.example.com"/>
</root>

And

<bort:root xmlns:bort="urn:test.example.com"/>
</bort:root>

Since the qualified name of the element is {urn:test.example.com}root

At least that's how libxml2 and python seem to handle it.

@Thoro

This comment has been minimized.

Copy link

@Thoro Thoro commented Dec 2, 2019

As nimish said it shouldn't actually matter, but there's soo many broken XML implementations out there, that the support for custom prefix naming would be great. Just as suggested it should be a Marshal Time configuration.

A possible implementation could look like:

namespaces := map[string]string{
    "bort": "urn:test.example.com",
}

encoder := xml.NewEncoder(memWriter)
encoder.SetNamespaces(namespaces)
encoder.Encode(response)

Any namespace defined are defined at the top most location and then not issued at any sub location anymore. Could fix a lot of issue where you have to interface with bad implementations.

Edit: Interestingly the xmlns handling is implemented for attributes but not for elements! (see https://golang.org/src/encoding/xml/marshal.go createAttrPrefix)

@nimish

This comment has been minimized.

Copy link

@nimish nimish commented Dec 2, 2019

Yeah, unfortunately the real world is full of bad xml handling :(
Also, if two elements share a namespace, and one is a child of the other, it'll repeat the xmlns="" declaration leading to a lot of redundancy. Would be better to omit, or assign a prefix and use that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.