Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

encoding/xml: support for XML namespace prefixes #9519

Open
jordan2175 opened this issue Jan 6, 2015 · 26 comments
Open

encoding/xml: support for XML namespace prefixes #9519

jordan2175 opened this issue Jan 6, 2015 · 26 comments

Comments

@jordan2175
Copy link

@jordan2175 jordan2175 commented Jan 6, 2015

Marshal-ing data back to XML does not seem to support namespace prefixes.. For example:

https://play.golang.org/p/6CY71H7mb4

The input XML is: <stix:STIX_Package>, but when it writes it back out it does <STIX_Package xmlns="stix">

Also, there does not appear to be any data elements like xml.Name for adding namespaces to a struct.... Something maybe like xml.NS????

@mikioh mikioh changed the title Support for XML namespace prefixes encoding/xml: Support for XML namespace prefixes Jan 6, 2015
@mikioh
Copy link
Contributor

@mikioh mikioh commented Jan 6, 2015

Dup of #6800

@mikioh mikioh closed this Jan 6, 2015
@jordan2175
Copy link
Author

@jordan2175 jordan2175 commented Jan 6, 2015

I think what I am looking for, as I do more research on it, is for xml.Marshal to support prefixes

@jordan2175
Copy link
Author

@jordan2175 jordan2175 commented May 29, 2015

Here is some more details about this issue: If you run this code:

https://play.golang.org/p/EucDh59yiB

package main

import (
    "encoding/xml"
    "fmt"
)

type HouseType struct {
    XMLName   xml.Name `xml:"prefix11 House"`
    MessageId string   `xml:"message_id,attr"`
}

func main() {

    var tm HouseType
    tm.MessageId = "test1234"

    var data1 []byte
    data1, _ = xml.MarshalIndent(tm, "", "    ")

    fmt.Println("Marshal")
    fmt.Println(string(data1))

    rawxml := `<prefix11:House message_id="1466" in_response_to="1"></prefix11:House>`

    var tm2 HouseType
    xml.Unmarshal([]byte(rawxml), &tm2)
    fmt.Println("\nUnmarshal")
    fmt.Println("Message ID", tm2.MessageId)
}

It will print out:

Marshal
<House xmlns="prefix11" message_id="test1234"></House>

Unmarshal
Message ID 1466

You will see that the marshal command does not make use of the prefix correctly. It should be prefix11:House. If I change the struct to look like the following then the marshal command will work but the unmarshal will not. NOTE the ":" in the XMLName field. So I can either marshal or unmarshal, but not both with prefixed XML

https://play.golang.org/p/44CMHXb3YM

type HouseType struct {
    XMLName   xml.Name `xml:"prefix11:House"`
    MessageId string   `xml:"message_id,attr"`
}

Making this change, changes the output to:

Marshal
<prefix11:House message_id="test1234"></prefix11:House>

Unmarshal
Message ID
@rsc
Copy link
Contributor

@rsc rsc commented Jul 23, 2015

See #11841.

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 23, 2015

CL https://golang.org/cl/12570 mentions this issue.

rsc added a commit that referenced this issue Jul 27, 2015
There is clearly work to do here with respect to xml name spaces,
but I don't believe the changes in this cycle are clearly correct.
The changes in this cycle have visible impact on the generated xml,
possibly breaking existing programs, and yet it's not clear that they
are the end of the story: there is still significant confusion about how
name spaces work or should work (see #9519, #9775, #8167, #7113).

I would like to wait to make breaking changes until we completely
understand what the behavior should be and can evaluate the benefit
of those breaking changes. My main concern here is that we will break
programs in Go 1.5 for the sake of name space adjustments and then
while trying to fix those other bugs we'll break programs in Go 1.6 too.
Let's wait until we know all the changes we want to make before we
decide whether or how to break existing programs.

This CL reverts:

5ae822b encoding/xml: minor changes
bb7e665 encoding/xml: fix xmlns= behavior
9f9d66d encoding/xml: fix default namespace of tags
b69ea01 encoding/xml: fix namespaces in a>b tags
3be158d encoding/xml: encoding name spaces correctly

and adjusts tests from

a9dddb5 encoding/xml: add more EncodeToken tests.

to expect Go 1.4 behavior.

I have confirmed that the name space parts of the test suite
as of this CL passes against the Go 1.4 encoding/xml package,
indicating that this CL successfully restores the Go 1.4 behavior.

(Other tests do not, but that's because there were some real
bug fixes in this cycle that are being kept. Specifically, the
tests that don't pass in Go 1.4 are TestMarshal's NestedAndComment
case, TestEncodeToken's encoding of newlines, and
TestSimpleUseOfEncodeToken returning an error for invalid
token types.)

I also checked that the Go 1.4 tests pass when run against
this copy of the sources.

Fixes #11841.

Change-Id: I97de06761038b40388ef6e3a55547ff43edee7cb
Reviewed-on: https://go-review.googlesource.com/12570
Reviewed-by: Nigel Tao <nigeltao@golang.org>
@rsc rsc changed the title encoding/xml: Support for XML namespace prefixes encoding/xml: support for XML namespace prefixes Nov 5, 2015
@rsc
Copy link
Contributor

@rsc rsc commented Nov 25, 2015

Blocked on #13400.

@rsc rsc modified the milestones: Go1.7, Go1.6 Nov 25, 2015
@rsc rsc modified the milestones: Go1.8, Go1.7 May 18, 2016
@rothskeller
Copy link

@rothskeller rothskeller commented Jun 14, 2016

Just a note in support of this proposal: with encoding/xml as it stands, it is infeasible to interact with SOAP web services that use digitally signed requests. This is because XML digital signatures require the XML to be in canonical form, which requires a greater degree of control over namespaces than encoding/xml provides. Admittedly, calling a SOAP service from a Go program is kind of like towing a trailer behind your Ferrari, but sometimes interoperability with legacy systems is essential.

@vania-pooh
Copy link

@vania-pooh vania-pooh commented Oct 7, 2016

Just a workaround while waiting for a fix:

type Data ...

type Root struct {
    XMLName  xml.Name  `xml:"prefix:root"`
    XmlNS    string    `xml:"xmlns:prefix,attr"`
    Data Data `xml:"data"`
}

root := Root {
    XmlNS: "urn:test.example.com",
    Data: ...,
}

b, err := xml.MarshalIndent(root, "", "    ")

This will produce the following:

<prefix:root xmlns:prefix="urn:test.example.com"/>
    <data>...</data>
</prefix:root>
@quentinmit quentinmit added the NeedsFix label Oct 7, 2016
@rsc rsc added this to the Go1.11 milestone Nov 22, 2017
Zauberstuhl added a commit to Zauberstuhl/go that referenced this issue Jan 20, 2018
Zauberstuhl added a commit to Zauberstuhl/go that referenced this issue Jan 22, 2018
@Zauberstuhl
Copy link

@Zauberstuhl Zauberstuhl commented Jan 22, 2018

Hi,

since I really wanted this in my private project I started playing around and
I actually have a working version for me. Could someone with deeper knowledge of the lib take a look at the changes and tell me whether that actually makes sense (Zauberstuhl@eb84a70)?

Here is an example of how that could look like:
https://play.golang.org/p/O4Ene-8GrVV

Cheers

@lkzcover
Copy link

@lkzcover lkzcover commented Feb 12, 2018

@sophos this works for XML-tags, but do not work for XML-attributes

@gopherbot
Copy link

@gopherbot gopherbot commented Jun 4, 2018

Change https://golang.org/cl/116056 mentions this issue: encoding/xml: fix printing of namespace prefix in tag names

@iWdGo
Copy link
Contributor

@iWdGo iWdGo commented Jun 4, 2018

With the submitted fix, prefix displays when defined in tag names, i.e. no made up prefix is taken into account. The URL of the name space is also returned for an End Token as documentation requires.
Translating the prefix is popping the NS which is unavailable for translate. Translate for an End Token has been moved inside the pop element part.

When a tag.Space has no prefix, it is the default space xmlns=".Space" according to documentation. Since there is no prefix the print remains <tag.Name.local … and not <Space:.Local…

The fix works on top of the merged fixes of the list issues in #13400 as namespace standard needs to be enforced before improving handling.

@Nerzal
Copy link

@Nerzal Nerzal commented Oct 10, 2018

Just a workaround while waiting for a fix:

type Data ...

type Root struct {
    XMLName  xml.Name  `xml:"prefix:root"`
    XmlNS    string    `xml:"xmlns:prefix,attr"`
    Data Data `xml:"data"`
}

root := Root {
    XmlNS: "urn:test.example.com",
    Data: ...,
}

b, err := xml.MarshalIndent(root, "", "    ")

This will produce the following:

<prefix:root xmlns:prefix="urn:test.example.com"/>
    <data>...</data>
</prefix:root>

This workaround only works when marshaling, but fails when you want to unmarshal :/

@genez
Copy link

@genez genez commented Nov 8, 2018

I have the same issue, while interacting with an Italian government service.

This playground demonstrate the issue: https://play.golang.org/p/H5Hibbci81_n

@lelvisl
Copy link

@lelvisl lelvisl commented May 7, 2019

@nimish
Copy link

@nimish nimish commented Nov 30, 2019

For the record, the prefix shouldn't actually matter. If two prefixes resolve to the same namespace then they are identical, including the empty prefix.

https://www.w3.org/TR/xml-names/#NT-PrefixedName

Note that the prefix functions only as a placeholder for a namespace name. Applications SHOULD use the namespace name, not the prefix, in constructing names whose scope extends beyond the containing document.

The full solution here is for the package to support full qualified names, and then have some notion of a map between Namespaces and Prefixes that is determined at marshal-time.

<prefix:root xmlns:prefix="urn:test.example.com"/>
</prefix:root>

Is (or should be) treated identically to:

<root xmlns="urn:test.example.com"/>
</root>

And

<bort:root xmlns:bort="urn:test.example.com"/>
</bort:root>

Since the qualified name of the element is {urn:test.example.com}root

At least that's how libxml2 and python seem to handle it.

@Thoro
Copy link

@Thoro Thoro commented Dec 2, 2019

As nimish said it shouldn't actually matter, but there's soo many broken XML implementations out there, that the support for custom prefix naming would be great. Just as suggested it should be a Marshal Time configuration.

A possible implementation could look like:

namespaces := map[string]string{
    "bort": "urn:test.example.com",
}

encoder := xml.NewEncoder(memWriter)
encoder.SetNamespaces(namespaces)
encoder.Encode(response)

Any namespace defined are defined at the top most location and then not issued at any sub location anymore. Could fix a lot of issue where you have to interface with bad implementations.

Edit: Interestingly the xmlns handling is implemented for attributes but not for elements! (see https://golang.org/src/encoding/xml/marshal.go createAttrPrefix)

@nimish
Copy link

@nimish nimish commented Dec 2, 2019

Yeah, unfortunately the real world is full of bad xml handling :(
Also, if two elements share a namespace, and one is a child of the other, it'll repeat the xmlns="" declaration leading to a lot of redundancy. Would be better to omit, or assign a prefix and use that.

@lelvisl
Copy link

@lelvisl lelvisl commented Feb 20, 2020

any updates?

achilleas-k added a commit to achilleas-k/libgin that referenced this issue Mar 10, 2020
It seems that unmarshaling XML data with attributes that have custom
namespace prefixes isn't fully supported:
golang/go#9519

Adjusted the top-level tag's (resource) namespace attribute to be
included in the XMLName field's tags.  More importantly, there's a new
DataCite method for fixing the schema attributes, since they're not
properly loaded when reading existing XML files.

Also, a new loader function takes a filename and loads the data into a
new DataCite struct, then runs the FixSchemaAttrs() method before
returning the loaded structure.
achilleas-k added a commit to achilleas-k/libgin that referenced this issue Mar 10, 2020
It seems that unmarshaling XML data with attributes that have custom
namespace prefixes isn't fully supported:
golang/go#9519

Adjusted the top-level tag's (resource) namespace attribute to be
included in the XMLName field's tags.  More importantly, there's a new
DataCite method for fixing the schema attributes, since they're not
properly loaded when reading existing XML files.

Also, a new loader function takes a filename and loads the data into a
new DataCite struct, then runs the FixSchemaAttrs() method before
returning the loaded structure.
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jul 8, 2020

Any updates will be reported here. Please don't ask for updates here. Ask on a forum instead. See https://golang.org/wiki/Questions. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.