Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

encoding/xml: XML namespace prefixes not fully supported #6800

Closed
gopherbot opened this Issue Nov 20, 2013 · 35 comments

Comments

Projects
None yet
@gopherbot
Copy link

gopherbot commented Nov 20, 2013

by anthonyeden:

It is possible to create a prefix on elements by simply defining the prefix as part of
the name in the XMLName declaration, but the xmlns attribute is output without the
prefix. Example: http://play.golang.org/p/KeIHB4_z5L

Thus it would seem that the encoding/xml package can only deal with document-level
namespaces, not nested namespaces, is this correct?
@rsc

This comment has been minimized.

Copy link
Contributor

rsc commented Nov 27, 2013

Comment 1:

Labels changed: added go1.3maybe.

@rsc

This comment has been minimized.

Copy link
Contributor

rsc commented Dec 4, 2013

Comment 2:

Labels changed: added release-none, removed go1.3maybe.

@rsc

This comment has been minimized.

Copy link
Contributor

rsc commented Dec 4, 2013

Comment 3:

Labels changed: added repo-main.

@anacrolix

This comment has been minimized.

Copy link
Contributor

anacrolix commented Dec 10, 2013

Comment 4:

This generates:
<example:foo xmlns="urn:example"></example:foo>
Am I correct in believing you require:
<example:foo xmlns:example="urn"></example:foo>
?
@gopherbot

This comment has been minimized.

Copy link
Author

gopherbot commented Dec 10, 2013

Comment 5 by anthonyeden:

Yes, you are correct.
@gopherbot

This comment has been minimized.

Copy link
Author

gopherbot commented Dec 10, 2013

Comment 6 by anthonyeden:

FWIW, I'm creating an RFC5730-RFC5733 implementation in Go, which is what triggered this
requirement.
@anacrolix

This comment has been minimized.

Copy link
Contributor

anacrolix commented Jan 4, 2014

@anacrolix

This comment has been minimized.

Copy link
Contributor

anacrolix commented Jan 4, 2014

Comment 8:

And https://bitbucket.org/anacrolix/dms/issue/25/get-soap-responses-to-work-with-upnp.
Perhaps this will help explain the shortcomings here.
@gopherbot

This comment has been minimized.

Copy link
Author

gopherbot commented Nov 8, 2014

Comment 9 by camilo.aguilar:

The lack of good support for namespace prefixes does seem to be affecting us here too
hooklift/gowsdl#17
@jordan2175

This comment has been minimized.

Copy link

jordan2175 commented Jan 6, 2015

As I commented on #9519 can we get the ability for xml.Marshal to use prefixes?

@rogpeppe

This comment has been minimized.

Copy link
Contributor

rogpeppe commented Feb 25, 2015

@rogpeppe rogpeppe closed this Feb 25, 2015

@mikioh mikioh added this to the Go1.5 milestone Feb 25, 2015

@jordan2175

This comment has been minimized.

Copy link

jordan2175 commented Feb 28, 2015

Can we verify that it will in fact work with an XML Namespace aspect like (note the various XMLNS prefixes “stix:, indicator:, cybox:”):

<stix:STIX_Package xsi:schemaLocation="
http://cybox.mitre.org/common-2 http://cybox.mitre.org/XMLSchema/common/2.1/cybox_common.xsd
http://cybox.mitre.org/cybox-2 http://cybox.mitre.org/XMLSchema/core/2.1/cybox_core.xsd
http://cybox.mitre.org/default_vocabularies-2
http://cybox.mitre.org/XMLSchema/default_vocabularies/2.1/cybox_default_vocabularies.xsd
http://cybox.mitre.org/objects#URIObject-2 http://cybox.mitre.org/XMLSchema/objects/URI/2.1/URI_Object.xsd
http://stix.mitre.org/Indicator-2 http://stix.mitre.org/XMLSchema/indicator/2.1.1/indicator.xsd
http://stix.mitre.org/common-1 http://stix.mitre.org/XMLSchema/common/1.1.1/stix_common.xsd
http://stix.mitre.org/default_vocabularies-1 http://stix.mitre.org/XMLSchema/default_vocabularies/1.1.1/stix_default_vocabularies.xsd
http://stix.mitre.org/stix-1 http://stix.mitre.org/XMLSchema/core/1.1.1/stix_core.xsd"
id="example:Package-8fab937e-b694-11e3-b71c-0800271e87d2"
timestamp="2014-05-08T09:00:00.000000Z"
version="1.1.1">

<stix:Indicators>
    <stix:Indicator id="example:Indicator-d81f86b9-975b-bc0b-775e-810c5ad45a4f" xsi:type="indicator:IndicatorType">
        <indicator:Title>Malicious site hosting downloader</indicator:Title>
        <indicator:Type xsi:type="stixVocabs:IndicatorTypeVocab-1.0">URL Watchlist</indicator:Type>
        <indicator:Observable id="example:Observable-ee59c28e-d922-480e-9b7b-a79502696505">
            <cybox:Object id="example:URI-b13ae3fc-80af-49c2-9de9-f713abc070ba">
                <cybox:Properties xsi:type="URIObj:URIObjectType" type="URL">
                    <URIObj:Value condition="Equals">http://x4z9arb.cn/4712</URIObj:Value>
                </cybox:Properties>
            </cybox:Object>
        </indicator:Observable>
    </stix:Indicator>
</stix:Indicators>

/stix:STIX_Package

Thanks,
Bret
PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."

On Feb 25, 2015, at 11:03 AM, Roger Peppe <notifications@github.com mailto:notifications@github.com> wrote:

Fixed by https://go-review.googlesource.com/2660 https://go-review.googlesource.com/2660

Reply to this email directly or view it on GitHub #6800 (comment).

@rogpeppe

This comment has been minimized.

Copy link
Contributor

rogpeppe commented Mar 1, 2015

I'm sorry, I don't quite understand what you mean when you ask whether "it will in fact work with an XML Namespace aspect like ...".

@jordan2175

This comment has been minimized.

Copy link

jordan2175 commented Mar 2, 2015

Given the XML example, can your patch parse it correctly and then re-marshall it back in to the same thing? I just want to make sure that it works with all of the namespace prefixes. Thanks SO MUCH for the patch and for fixing this issue.

Thanks,
Bret
PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."

On Mar 1, 2015, at 1:58 AM, Roger Peppe notifications@github.com wrote:

I'm sorry, I don't quite understand what you mean when you ask whether "it will in fact work with an XML Namespace aspect like ...".


Reply to this email directly or view it on GitHub #6800 (comment).

@davecheney

This comment has been minimized.

Copy link
Contributor

davecheney commented Mar 2, 2015

There are no xmlns attributes in the fragment you presented. That looks
like xml schema information.

On Mon, Mar 2, 2015 at 2:33 PM, jordan2175 notifications@github.com wrote:

Given the XML example, can your patch parse it correctly and then
re-marshall it back in to the same thing? I just want to make sure that it
works with all of the namespace prefixes. Thanks SO MUCH for the patch and
for fixing this issue.

Thanks,
Bret
PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that
can not be unscrambled is an egg."

On Mar 1, 2015, at 1:58 AM, Roger Peppe notifications@github.com
wrote:

I'm sorry, I don't quite understand what you mean when you ask whether
"it will in fact work with an XML Namespace aspect like ...".


Reply to this email directly or view it on GitHub <
https://github.com/golang/go/issues/6800#issuecomment-76588584>.


Reply to this email directly or view it on GitHub
#6800 (comment).

@jordan2175

This comment has been minimized.

Copy link

jordan2175 commented Mar 2, 2015

Here is an example with them.

<stix:STIX_Package
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:stix="http://stix.mitre.org/stix-1"
xmlns:indicator="http://stix.mitre.org/Indicator-2"
xmlns:cybox="http://cybox.mitre.org/cybox-2"
xmlns:AddressObject="http://cybox.mitre.org/objects#AddressObject-2"
xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2"
xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1"
xmlns:ExampleNamespace="http://example.com/"
xsi:schemaLocation="
http://stix.mitre.org/stix-1 http://stix.mitre.org/XMLSchema/core/1.0.1/stix_core.xsd
http://stix.mitre.org/Indicator-2 http://stix.mitre.org/XMLSchema/indicator/2.0.1/indicator.xsd
http://cybox.mitre.org/default_vocabularies-2 http://cybox.mitre.org/XMLSchema/default_vocabularies/2.0.1/cybox_default_vocabularies.xsd
http://stix.mitre.org/default_vocabularies-1 http://stix.mitre.org/XMLSchema/default_vocabularies/1.0.1/stix_default_vocabularies.xsd
http://cybox.mitre.org/objects#AddressObject-2 http://cybox.mitre.org/XMLSchema/objects/Address/2.0.1/Address_Object.xsd"
id="ExampleNamespace:STIXPackage-33fe3b22-0201-47cf-85d0-97c02164528d"
version="1.0.1">
<stix:STIX_Header>
    <stix:Title>Example watchlist that contains IP information.</stix:Title>
    <stix:Package_Intent xsi:type="stixVocabs:PackageIntentVocab-1.0">Indicators - Watchlist</stix:Package_Intent>
</stix:STIX_Header>
<stix:Indicators>
    <stix:Indicator xsi:type="indicator:IndicatorType" id="ExampleNamespace:Indicator-33fe3b22-0201-47cf-85d0-97c02164528d">
        <indicator:Type xsi:type="stixVocabs:IndicatorTypeVocab-1.0">IP Watchlist</indicator:Type>
        <indicator:Description>Sample IP Address Indicator for this watchlist. This contains one indicator with a set of three IP addresses in the watchlist.</indicator:Description>
        <indicator:Observable  id="ExampleNamespace:Observable-1c798262-a4cd-434d-a958-884d6980c459">
            <cybox:Object id="ExampleNamespace:Object-1980ce43-8e03-490b-863a-ea404d12242e">
                <cybox:Properties xsi:type="AddressObject:AddressObjectType">
                    <AddressObject:Address_Value condition="Equals" apply_condition="ANY">10.0.0.0##comma##10.0.0.1##comma##10.0.0.2</AddressObject:Address_Value>
                </cybox:Properties>
            </cybox:Object>
        </indicator:Observable>
    </stix:Indicator>
</stix:Indicators>
</stix:STIX_Package>
@davecheney

This comment has been minimized.

Copy link
Contributor

davecheney commented Mar 2, 2015

I'm sorry, github ate your upload. Can you put it in a file an attach or
use a pastebin service ?

On Mon, Mar 2, 2015 at 2:58 PM, jordan2175 notifications@github.com wrote:

Silly of me... Here is an example with them.

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:stix="http://stix.mitre.org/stix-1"
xmlns:indicator="http://stix.mitre.org/Indicator-2"
xmlns:cybox="http://cybox.mitre.org/cybox-2"
xmlns:AddressObject="http://cybox.mitre.org/objects#AddressObject-2"
xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2"
xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1"
xmlns:ExampleNamespace="http://example.com/"
xsi:schemaLocation="
http://stix.mitre.org/stix-1
http://stix.mitre.org/XMLSchema/core/1.0.1/stix_core.xsd
http://stix.mitre.org/Indicator-2
http://stix.mitre.org/XMLSchema/indicator/2.0.1/indicator.xsd
http://cybox.mitre.org/default_vocabularies-2
http://cybox.mitre.org/XMLSchema/default_vocabularies/2.0.1/cybox_default_vocabularies.xsd
http://stix.mitre.org/default_vocabularies-1
http://stix.mitre.org/XMLSchema/default_vocabularies/1.0.1/stix_default_vocabularies.xsd
http://cybox.mitre.org/objects#AddressObject-2
http://cybox.mitre.org/XMLSchema/objects/Address/2.0.1/Address_Object.xsd"
id="ExampleNamespace:STIXPackage-33fe3b22-0201-47cf-85d0-97c02164528d"
version="1.0.1">
stix:STIX_Header
stix:TitleExample watchlist that contains IP information./stix:Title
Indicators - Watchlist/stix:Package_Intent
/stix:STIX_Header
stix:Indicators

IP Watchlist/indicator:Type
indicator:DescriptionSample IP Address Indicator for this watchlist. This
contains one indicator with a set of three IP addresses in the watchlist.
/indicator:Description

10.0.0.0##comma##10 #10
#10 #10
/AddressObject:Address_Value
/cybox:Properties
/cybox:Object
/indicator:Observable
/stix:Indicator
/stix:Indicators
/stix:STIX_Package


Reply to this email directly or view it on GitHub
#6800 (comment).

@jordan2175

This comment has been minimized.

Copy link

jordan2175 commented Mar 2, 2015

I see that. I edited the post above to fix it. I also copied it to pastebin,here
http://pastebin.com/kbtDUWdE

@jordan2175

This comment has been minimized.

Copy link

jordan2175 commented Mar 3, 2015

Here is the example from their website as well.

http://stix.mitre.org/about/example_watchlist.html http://stix.mitre.org/about/example_watchlist.html

Thanks,
Bret
PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."

On Mar 1, 2015, at 9:04 PM, Dave Cheney notifications@github.com wrote:

I'm sorry, github ate your upload. Can you put it in a file an attach or
use a pastebin service ?

On Mon, Mar 2, 2015 at 2:58 PM, jordan2175 notifications@github.com wrote:

Silly of me... Here is an example with them.

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:stix="http://stix.mitre.org/stix-1"
xmlns:indicator="http://stix.mitre.org/Indicator-2"
xmlns:cybox="http://cybox.mitre.org/cybox-2"
xmlns:AddressObject="http://cybox.mitre.org/objects#AddressObject-2"
xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2"
xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1"
xmlns:ExampleNamespace="http://example.com/"
xsi:schemaLocation="
http://stix.mitre.org/stix-1
http://stix.mitre.org/XMLSchema/core/1.0.1/stix_core.xsd
http://stix.mitre.org/Indicator-2
http://stix.mitre.org/XMLSchema/indicator/2.0.1/indicator.xsd
http://cybox.mitre.org/default_vocabularies-2
http://cybox.mitre.org/XMLSchema/default_vocabularies/2.0.1/cybox_default_vocabularies.xsd
http://stix.mitre.org/default_vocabularies-1
http://stix.mitre.org/XMLSchema/default_vocabularies/1.0.1/stix_default_vocabularies.xsd
http://cybox.mitre.org/objects#AddressObject-2
http://cybox.mitre.org/XMLSchema/objects/Address/2.0.1/Address_Object.xsd"
id="ExampleNamespace:STIXPackage-33fe3b22-0201-47cf-85d0-97c02164528d"
version="1.0.1">
stix:STIX_Header
stix:TitleExample watchlist that contains IP information./stix:Title
Indicators - Watchlist/stix:Package_Intent
/stix:STIX_Header
stix:Indicators

IP Watchlist/indicator:Type
indicator:DescriptionSample IP Address Indicator for this watchlist. This
contains one indicator with a set of three IP addresses in the watchlist.
/indicator:Description

10.0.0.0##comma##10 #10
#10 #10
/AddressObject:Address_Value
/cybox:Properties
/cybox:Object
/indicator:Observable
/stix:Indicator
/stix:Indicators
/stix:STIX_Package


Reply to this email directly or view it on GitHub
#6800 (comment).


Reply to this email directly or view it on GitHub #6800 (comment).

@c4milo

This comment has been minimized.

Copy link
Member

c4milo commented Mar 18, 2015

@rogpeppe, did this fix land in Go 1.4.x?

@c4milo

This comment has been minimized.

Copy link
Member

c4milo commented Mar 18, 2015

@rogpeppe, I'm having problems marshaling, when attempting to consume a webservice that is expecting namespaces declared at the top along with prefixes, and each inner element using the prefix as opposed to the actual namespace attribute. You can find more details about this issue here: hooklift/gowsdl#17

@notzippy

This comment has been minimized.

Copy link

notzippy commented Mar 18, 2015

@c4milo This fix is not in go 1.4.1

@jordan2175

This comment has been minimized.

Copy link

jordan2175 commented Mar 18, 2015

Here is another example that the current XML bindings can not process.. It is a bit different than the current issues, I think, or maybe it is related. If I take out all but one of the xmlns attributes, then it works.

<taxii_11:Discovery_Response 
    xmlns:taxii="http://taxii.mitre.org/messages/taxii_xml_binding-1"
    xmlns:taxii_11="http://taxii.mitre.org/messages/taxii_xml_binding-1.1"
    xmlns:tdq="http://taxii.mitre.org/query/taxii_default_query-1" 
    message_id="32898" 
    in_response_to="1">
</taxii_11:Discovery_Response>
@wookoouk

This comment has been minimized.

Copy link

wookoouk commented Mar 18, 2015

I am trying to parse (small snippet):

<w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"></w:document>

but I get: error: expected element type <w:document> but have

I could change

XMLName xml.Name `xml:"w:document"`

to

XMLName xml.Name `xml:"document"`

but that breaks everything else.

@c4milo

This comment has been minimized.

Copy link
Member

c4milo commented Mar 18, 2015

@wookoouk this fix is going to land in Go 1.5. So, we are going to still see this failing in Go 1.4.x

@wookoouk

This comment has been minimized.

Copy link

wookoouk commented Mar 18, 2015

Is it available in HEAD?

@c4milo

This comment has been minimized.

Copy link
Member

c4milo commented Mar 18, 2015

@wookoouk yes, master branch, aka tip, should have it.

@wookoouk

This comment has been minimized.

Copy link

wookoouk commented Mar 18, 2015

Cheers @c4milo

@notzippy

This comment has been minimized.

Copy link

notzippy commented Mar 18, 2015

I found that the master did not make much difference, I used another package https://github.com/jteeuwen/go-pkg-xmlx to read the xml, correct the name space, and output the xml

@jordan2175

This comment has been minimized.

Copy link

jordan2175 commented Mar 18, 2015

To everyone that is writing the code / patches and everyone that is contributing examples of things that are not working, THANKS!!!! What a great community. Lets make Golang the best!

@wookoouk

This comment has been minimized.

Copy link

wookoouk commented Mar 18, 2015

yep, sadly the same for me:

go version devel +6ffed30 Wed Mar 18 15:14:37 2015 +0000 darwin/amd64
[xmd] go run xmd.go tomd test.docx                                                                       16:58:32  ☁  master ☂ ✖ ⚡ ✭
word/document.xml
error: expected element type <w:document> but have <document>%
@c4milo

This comment has been minimized.

Copy link
Member

c4milo commented Mar 18, 2015

This issue should be re-opened then.

@mikioh

This comment has been minimized.

Copy link
Contributor

mikioh commented Mar 19, 2015

Please open a new issue if necessary. The above additional information confuses the issue whether the applied fix generates a new issue (breaks existing code and we should revert the fix) or just not fixing undiscovered, already reported issues.

If you are saying that the applied change 2660 should be reverted for some reason, please write on this issue so.
If you are saying that the applied change 2660 does not fix your issue, please search existing issues and write additional information for contributors, or open a new issue if necessary.

Running issue filters with "encoding/xml in:title" still reports there're 19 open issues.

@jordan2175

This comment has been minimized.

Copy link

jordan2175 commented Mar 19, 2015

I do not believe this issue has actually been fixed with patch 2660. Especially when you consider the fact that issues #9519 and #9775 were closed and folded in to this issue and neither one of them has been addressed. That means until they are also fixed, this issue is not done and should be reopened.

@mikioh

This comment has been minimized.

Copy link
Contributor

mikioh commented Mar 19, 2015

@jordan2175,

Fine, re-open both #9519 and #9775 instead. But please add more useful information for contributors in #9519 and #9775. As @rogpeppe said in above:

I'm sorry, I don't quite understand what you mean when you ask whether "it will in fact work with an XML Namespace aspect like ...".
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.