never mind. this is the in context parsing that just needs to be ported.

Added test coverage, and decorators are working find on fragments, their nodes and nodesets.

Ah! Looks like children() is not properly decorated. Looks like we could do some cleaning up of node set decorating in general.

I'll open a new ticket. Issues collaborator fail. Again. Sigh.

Not at this point.

In the future, if you have a questions (such as this) rather than bugs, please send an email to the mailing list. Thanks.

Below is the mailing list address:

http://groups.google.com/group/nokogiri-talk

Ya, never mind, he should use the Node constants.

This sounds like a great feature to add to your client application. I think this can be accomplished without being in mainline nokogiri. Please implement it, and open a new ticket when you feel it's ready to be merge to the main line.

I don't know how a feature that improves Nokogiri could just be useful for me, specially when it tries to fix real issues in fragment management.

For sure I will try to code it. I just would like to ask for some help or tip, specially since I don't know how feasible is to insert verbatim a raw XML string into the string generated by Document#to_xml.
I would really appreciate any help on it.

PS: I forgot to add a link to a test_unit which shows the errors I commented in this bug report:

http://gist.github.com/252738

Should I open a different report for it?
Thanks.

never mind. #109 is reopened.

Added test unit: http://gist.github.com/252531

This is a known issue. attributes is a convenience method that works well for most people. If you're dealing with namespaced attributes, you must use the attribute_nodes method.

Thanks, I understand and IMHO it makes sense (as a node with two attributes sharing name but different prefix is very exotic).
However I suggest to clarify it in the documentation.
Regards.

No. This is what DTD validation is for. Nodes with no namespace are perfectly valid. You need to validate your document against a DTD before processing if you're having that problem.

ok, understood. Thanks.

Let me correct myself when I said:

"So IMHO what Nokogiri should do is very easy: just nothing. When inserting a node Nokogiri shouldn't check the node namespace prefix, neither try to guess the appropriate one, neither replace it."

I understand that this is not possible as Nokogiri requires all the nodes having correct NS prefixes (this is, prefixes matching the node namespace bindings). So it's not possible for Nokogiri to insert a node with unknown NS prefix into the document.

So IMHO what Nokogiri should do is:

If the new node has NS prefixes then check that the parent node (and not the doc root node) namespaces definitions contains these prefixes:

a) If so, insert the new node verbatim and that's all.

b) If new node prefixes (or absense of them) don't match parent node NS definitions (including a possible default NS for this parent node) then Nokogiri could do the following:

b1) Remove the new node prefixes (as it does now). However in a XML in which NS is required for all the nodes this would just work if the parent node (and not the root node) has a default namespace (and anyhow this doesn't ensure that the XML would be valid when checking its XML schema).

b2) Raise an exception. It should just occur if the document doesn't allow nodes without NS (it could be default NS in the parent node so no prefix is required).

I really like (and need) option "b2" working as in my project I work with strict XML documents requiring namespace por all the nodes. However for this to work it would be required a new parsing/document option like "REQUIRE_NS_FOR_ALL_NODES".

So when "REQUIRE_NS_FOR_ALL_NODES" isenabled:

If the new node has NS prefixes then check that the parent node (and not the doc root node) namespaces definitions contains these prefixes:
a) If so, insert the new node verbatim and that's all.
b) If not, raise an exception (like "WrongNamespacePrefix").

If the new node has no NS prefixes then check that the parent node (and not the doc root node) namespaces definitions contains a default namespace.
a) If so, insert the new node verbatim and that's all.
b) If not, raise an exception (like "WrongNamespacePrefix").

When "REQUIRE_NS_FOR_ALL_NODES" is dissabled (as the current Nokogiri's behavior) then:

If the new node has NS prefixes then check that the parent node (and not the doc root node) namespaces definitions contains these prefixes:
a) If so, insert the new node verbatim and that's all.
b) If not, remove the prefixes. But never attemp to replace the prefix with others.

If the new node has no NS prefixes then check that the parent node (and not the doc root node) namespaces definitions contains a default namespace.
a) If so, insert the new node verbatim and that's all (Nokogiri FAILS in this point as I described in the report!!!).

Again, I strongly suggest never to replace or add a NS prefix into the new node. Inserting a valid node is responsability of the user. Also, Nokogiri FAILS when adding a NS prefix as reported above.

Hi,

The Nokogiri core developers practice "test driven development". This
means that, when we want to change Nokogiri's behavior, either by
adding a feature or fixing a bug, we first write a test (or tests)
that completely specifies the new behavior. You can see all of these
tests under the /test directory in Nokogiri's source respository.

We like very much to get bug reports and feature requests that include
one or both of the following:

1. An indication of which existing test(s) should be changed
   
2. New (failing) test(s), specifying the desired behavior

Or, less desirably, we would like to see clear, runnable sample code
that is easily convertible into a failing test.

The reason we like to see these tests in a bug report is because the
core developers (who maintain Nokogiri in their spare time, for free),
do not have to spend time trying to translate imprecise English into
Ruby code.

Secondarily, a test case indicates to us that the reporter has thought
through the issue completely, and cares enough about the issue to have
taken the time to present the argument clearly and concisely in the
lingua franca of the Ruby community (i.e., Ruby).

Why am I bringing this up now? This issue, so far, contains over 1100
words and no test cases. Although there are a few snippets of XML and
Ruby code, I do not consider any of them to be a complete runnable
example or test.

I do not doubt there is clear logic in your argument; I just don't
have the time to read it all and translate it into
requirements. Explaining your issue in Ruby tests instead of English
will be clearer, more dense and will better communicate your issue to
the core developers.

Thank you for reporting this issue. I am looking forward to seeing
your failing tests, and better understanding what you are asking the
core developers to build for you.

Cheers.

Ok, I've sent a mail to Nokogiri maillist containing a test_unit describing the issue I mean.
However the issue is not easy to describe just with unit cases so I strongly suggest, please, to read my above report after checking the test unit.

To simplify a lot all the report: Nokogiri should NEVER add/guess/replace the NS prefixes of a new inserted node. It's responsability of the user/client to use the appropriate ns prefixes.

test unit created: http://gist.github.com/250927

Thank you for the two test cases you submitted. I think I understand the first one, which states that a new fragment, without a namespace declared in it, should be given the default document namespace (if there is one).

I do not understand the second test case. It is nearly identical to the first, and specifies behavior in which a non-default namespace is applied to the fragment. Can you please explain why this behavior is expected?

I think I understand the first one, which states that a new fragment, without a namespace declared in it, should be given the default document namespace (if there is one).

And in case the document has no default namespace then the new node has no namespace binding, so adding parent node's namespace is wrong (or unexpected since Nokogiri has no way to determine the appropriate prefix, neither if the node must or not have a prefix).

I do not understand the second test case. It is nearly identical to the first, and specifies behavior in which a non-default namespace is applied to the fragment. Can you please explain why this behavior is expected?

The second case shows an error of the user who is trying to insert a node belonging to "urn:strint-rules" namespace using wrong prefix (no prefix in this case).
So from Nokogiri's point of view the new node belongs to parent node default namespace (I will comment this later as it's buggy IMHO) but Nokogiri adds "cp" prefix.

This is, the XML will get wrong since the user is inserting a wrong node (wrong prefix). Anyhow the XML should get as follow:

<?xml version="1.0" encoding="UTF-8"?>
<cp:ruleset xmlns="default.ns" xmlns:cp="urn:common-policy" xmlns:sr="urn:strint-rules">
  <cp:rule id="1"/>
  <cp:rule id="2"/>
  <sr:strict_rule name="sr1"/>
  <condition id="1" name="I belong to default namespace"/>
  <cp:rule id="3"/>
  <sr:strict_rule name="sr2"/>
  <strict_rule id="3" name="I'm wrong as I should have 'sr' prefix"/>
</cp:ruleset>

But Nokogiri adds parent node prefix "cp" which, as in case 1, is wrong.

You have not clarified the second test case at all. It appears to me that you are conflating DTD validation with the construction of a document.

Am I correct in assuming that you think Nokogiri should "know" that <strict_rule> belongs in the "sr" namespace because of some sort of DTD declaration?

No, I just say that Nokogiri should insert the new "strict_rule" node without prefix (as always) and its namespace binding should belong to "default.ns".

Yes, the resulting document would be incorrect according to DTD in case "default.ns" namespace has no element called "string_rule", but Nokogiri shouln't care of it at all.

I would like to add something important: After inspecting how Nokogiri inserts a new fragment (by doing 'doc.fragment' and so) I must say that it's 100% wrong.

Nokogiri tries to detect the new node namespace bindings by running "doc.fragment(new_node)" and it just inspects root node namespace bindings. This is wrong. It's not possible to know the appropriate bindings for a new node until we already know the exact parent node in which it will be inserted.

So IMHO Document#fragment method should entirely dissapear and instead Node#fragment should exist. Then the way to insert a node would be:

1) parent_node = xml.xpath("/ns1:root/ns1:child/ns2:list", {"ns1"=>....
2) frag =  parent_node.fragment(new_node)
3) parent_node.add_child(frag)

In point 1 we get the parent node in which we want to insert the new node.
In point 2 Nokogiri inspects the appropriate namespace bindings for the new node according to the parent node. To do it, the best way I know is by feching parent node namespaces with Xpath:

parent_node_ns = xml.xpath("/ns1:root/ns1:child/ns2:list/namespace::*", {"ns1"=>....

This returns an array of Namespace objects so then Nokogiri can inspect the new_node prefixes and match them against the list of Namespace objects.

Please note that taking the root node namespaces is completely wrong. For example take a looko to this XML:

<?xml version='1.0' encoding='UTF-8'?>
<root xmlns="urn:ns1">
  <ns2:child xmlns:ns2="urn:ns2" xmlns="urn:ns3">
    <elem name="I belong to 'urn:ns3'"/>
  </ns2:child>
</root>

• "root" node has no prefix but default ns "urn:ns1".
• "ns2:child" declares a new default ns for its children ("urn:ns3") which replaces the root node default namespace.
• So "elem" belongs to "urn:ns3" rather than "urn:ns1".

I've updated my test_unit and now it shows more cases, including two more to explain and prove what I said in my last comment:

http://gist.github.com/250927

Please rewrite these tests in the style you see in test/xml/test_document_fragment.rb.

I am more confused now than when we started. I am pretty sure that is not what you intended, so please try to explain your issue again, in a simpler / smaller / more concise example.

Ok, I must recognize that this issue is very complex as it gets deep into XML's most exotic cases when handling namespaces.

Let's start from the beginning. The following test_unit is simpler (I hope). Let me know your opinion. Thanks.

https://gist.github.com/2e0690803011227879e7

I have rewritten your tests, removing unnecessary markup and code (there was quite a bit that was unnecessary). I removed the last test, since it was redundant, and added a new test case for clarity (and to demonstrate how fragment namespaces currently work).

Please review https://gist.github.com/c149f6b74f811b6b93ae and let me know if these test cases accurately reflect your desired implementation.

Thanks, it's ok. However I've added one more test to your file:

http://gist.github.com/251573

The new test (last one) tries to probe than we should always speak about "parent node namespaces binding" rather than "root node namespaces" or "parent node namespaces".

Node namespaces bindings are got as follows:

• Take the namespaces of root node (level1).
• Take the namespaces of level2 and add them to the the previous list (replacing those matching the prefix).
• Take the namespaces of level3 and add them to the the previous list (replacing those matching the prefix).
• So we get the namespaces bindings for node level3 which is different than the namespaces declared in node level3.

Xpath and Libxml2 (and Nokogiri) allows getting node namespaces bindings by running:

doc.xpath("ns1:level1/ns2:level2/ns3:level3/namespace::*)

This gets an array of namespace definitions corresponding to level3 namespaces bindings, which is different than just the namespaces definitions existing in level3 (as explained before).

Hello. This last test is redundant, but I will include it if it
makes you happy.

You are explaining things in English again. I promise you, I am
not reading any of your explanations of functionality. I am only
reading what you code. Show me, don't tell me.

I am trying (and have been trying) to explain to you that the
spec code you are writing is your only means of communicating
changes in Nokogiri behavior to the core team. Test-driven
development: Development is driven by tests.

So, is https://gist.github.com/c149f6b74f811b6b93ae a complete
specification of the behavior that you desire? If so, then we can
start to discuss:

• whether this is desired behavior
• whether this is feasible to implement given current design and libxml limitations
• whether it conflicts with the current behavior, thereby possibly breaking existing applications

I hope, you are getting a sense of how the Nokogiri core
developers prefer to go about their work. I think you'll find
that most of the Ruby community does things this same way.

Please let me know your thoughts.

Well, I don't agree too much with the names you set to the tests:

• test_adding_a_fragment_should_use_the_document_default_namespace_when_root_is_nondefault
• test_adding_a_fragment_should_use_the_document_default_namespace_when_root_is_default
• test_adding_a_fragment_should_have_no_namespace_when_root_has_no_namespace

I just wanted to clarify that "document default namespace" (or "root node namespace" which has same meaning here) should NEVER be considered when inserting a new node, and instead "parent node namespaces bindings" should just be inspected (which is different than "parent node declared namespaces"). Just it. I'm sorry but I don't know how to clarify this with a test unit (this is what I tryed to explain with my lastest test though).

But if we agree on it then your test sounds good for me. So starting to discuss:

• Yes, this is the desired behavior.
• I expect this is feasible to implement as libxml2 (and Nokogiri) allows feching node namespaces bindings (using "namespace::") so Nokogiri can use this info to set the appropriate namespaces in the new inserted node. But for this, 'Document#fragment' should be removed and instead 'Node#fragment' should exist, since the new node namespaces bindings depends on its parent node* rather than on the root node (see "test_adding_a_fragment_should_use_parent_node_namespaces_rather_than_root_node_namespaces").
• I hope this doesn't break any existing application. It just fixes some corner cases (well, no so "corner") and IMHO nobody should expect a wrong result from Nokogiri.

I hope this is a good point to start. Regards.

I've created a new test unit just to probe that "node namespaces bindings" is diffetent than "node declared namespaces":
http://gist.github.com/251636

Yes, this is the desired behavior I expect this is feasible to implement

Sounds like you've got everything covered, and you don't need my advice. Please submit a patch when your implementation is complete and all tests pass.

Thanks for using Nokogiri.

Please, I just meant that I expect it to be feasible since libmlx2 and Nokogiri already implements fetching node namespaces correctly. I don't intend to say that it's easy to implement. I'm really sorry if my words weren't the most appropriate.

I can try to help in testing and specifications but I've no enough level to code such feature by myself. Please excuse my limited English.

ok, I'm already working on it. I've found where the issue is: in "xml/fragment_handler.rb" as it assumes it must inspect namespaces into document root. This is not valid since it's not possible to determine the namespaces fo a new node without knowing which will be its parent node.

So it's required a new method Node#fragment:

def fragment tags
  DocumentFragment.new(self, tags)
end

So I'm also modifying DocumentFragment and FragmentHandler so Nokogiri could get the namespaces binding for the parent node in which the element will be inserted.

I just have a doubt I don't know how to achieve:

Let's assume I've a Node element called "parent_node" and I want to get its namespaces bindings, this is, the same output as if I run:

doc.xpath(path_to_parent_node + "/namespace::*", ns)

So I would get an Array of XML::Namespace (please check http://gist.github.com/251636).>

How could I get the same Array just by having the Node "parent_node" rather than the xpath expression?
Please help me with this. I think I could fix the problem.
Thanks.

i will write a new method (which will require C code) to give you back the namespaces that are in scope for a node. In the meantime, though, you can use node.xpath("./namespace::*") as a substitute.

See branch 'namespaces' in tenderlove's repo. the method Node#namespaces returns a hash of all namespaces in scope for a node. Previously it returned only namespaces declared on the node.

That change will not be in 1.4.1, but will probably be in 1.4.2.

Thanks, I'm inspecting it right now.
I'm modyfing DocumentFragment so it creates a FragmentHandler with 3 parameters (instead of 2):

FragmentHandler.new(parent_node, self, tags)

Then in FragmentHandler#initialize I store 'parent_node' into @parent_node attribute.
Then I would use your new code to get namespaces bindings for the current parent node and use them to set the namespaces for each element in the new node.

I'll work on it tomorrow. Thanks a lot.

I think I've a working fix for this issue. Please check the branch "namespaces" in my forked Nokogiri:
http://github.com/ibc/nokogiri/commit/1fd50936f1d4d21172f0f8e1ea7f07c888691766

With this code, the following test units are passed:
http://gist.github.com/252440

Note that in these tests I use "Node#fragment" rather than "Document#fragment".
Previously, "Node#fragment" was an "alias" of "Document#fragment", but in my new code this method uses the current node (parent node) to get the namespace scopes and use them for the new fragment.

About the official test units, with the new code just one fails ("test_fragment_namespace_resolves_against_document_root"), but it works if we replace:

frag = doc.fragment

with:

frag = doc.root.fragment

The fact is that IMHO "doc.fragment" should be deprecated as fragments belong to nodes rather than documents.
A workaround wouuld be creating a "Document#fragment" method with just calls to "self.root.fragment".

Please let me know your opinion about the commit. Thanks a lot.

There is still a corner case in which inserting a node would fail: when the new fragment node also contains namespace declarations.
For this, it's required to insert the fragment top level node, get its namespace scopes and use them when inserting its children (and so on).
I'll work on it.

I've open a new report #189 with the suggestion of a different approach to insert a fragment.

Could this report be open again please? It's now "closed" but has activity and commited code :)

I've done a lot of improvements and now Nokogiri allows fragments containing namespace declarations and subnodes using them, and also subnodes containing namespace declaration and their subnodes using them. And also handling prefixed attributes with the namespaces declared into the fragment:
http://github.com/ibc/nokogiri/tree/namespaces

I've created a "total" test_unit for this issue:
http://gist.github.com/254109

There are 10 tests, some of them very exotic.
7 of them fail under current Nokogiri HEAD.
Just one fails under my fork and it fails due to a reported bug #192.

So if you can help me with bug #192 then I expect that inserting a complex fragment would work really well :)

Thanks.

The current code in my fork also fixes the bug #190.

Hi, have you developers had a chance to check my commit and test unit?
Is there any update or comment for this report?

If you need me to provide more data (along with the already provided test cases and patch) please ask it to me.

Thanks a lot.

PS: Could this report be open again please?

Invalid, pass -X+O

jruby -X+O -S cucumber features/book.feature

closing

Apparently Mike knows about this, so I'm closing it.

aliasing Node#next= and Node#previous= to Node#add*sibling(). closed by 6b6bf52.

Sorry, ruby was in 32 bits

closing.

XML::Node#add_child now accepts an optional +position+ argument. Closed by fec1b08.

this issue was addressed by the fix for #178.

added test coverage for fragments with leading entity. closed by fe0570b.

this is an issue with all versions of libxml2, at least back to 2.6.16

this is because a null byte is treated as a string terminator in C. you should work around this by removing null bytes from your document.

This is a bug in libxml2. We'll work with them to fix it, but there is nothing we can do about it in the nokogiri code base.

I know that. I've stumbled on this in the wild internets (some guy made a part of site in ms word and it had \0 in a non-standard attribute).

This also applies to other HTML -- I have observed it with anchors, i.e., "One line\nTwo line\n\n<a href="http://brokenlink.com">This won't be a link after parsing</a>"

However, if I wrap the text block in a <div> and </div>, it works. I'm thinking that the newline must somehow interfere with Nokogiri's ability to discern the insides as HTML.

OK, will investigate.

fixing leading text node with newline in fragment parsing. closed by b659302.

Great turnaround time, thanks a lot! Between this and the fix to the document.root.namespace exception you recently did, I'd love to see a gem bump soon so I can strip out my collection of kluge-fixes. :)

Should be bumped this weekend. Cross your fingers.

I think this is a bug in Ruby 1.9, but my commit to convert the exception to pure ruby seems to fix it:

f0675b3

Sorry, we can't tell libxml2 to do that. :-(

It looks like Stylesheet#serialize will honor the output method (it gives you back a string). If I made serialize deal with XSLT parameters, would that be acceptable?

Stylesheet#serialize doesn't actually apply a transform though does it? - it just serializes an XML document according to the output method declared by a stylesheet?
If that is the case, you could probably leave Stylesheet#serialize as is and simply redefine Stylesheet#apply_to to:

def apply_to document, params = []
  serialize(transform(document, params))
end

This kills two birds with one stone since apply_to already accepts params and applying the serialization suggested by the stylesheet rather than simply using "to_s" is probably better anyway?

Yes, you are correct! I like this solution a lot. I'll change it to do this.

using serialize in XSLT#apply_to so that xslt output methods are honored. closed by 84a2986

This was fixed here: e4976fd

and is a dup of #160

supporting :has() selectors. closed by af2856a

fixed docs for Node#attributes. Closed by 96a5436.

This has already been fixed in master, in commit b7eeeda.

Nokogiri::XML::Document should not define add_namespace().

Closed by e16be12

• lib/nokogiri/xml/document.rb removed add_namespace
• test/xml/test_document.rb

This was fixed.

I've pushed a branch named 'soap4r-bug' that reproduces this problem in the Nokogiri test suite.

adding tests for soap4r adapter and bugfixes. closed by ecdcb0a

This was fixed in 8111e7b

Figured it out!

http://gist.github.com/250477

Awesome, thanks. Now I just need to grok the XML catalog tomfoolery so I can stop libxml fetching the DTD over the net everytime I parse a file

What I've been doing, and I admit this probably isn't the "right way", is doing a "sub" on my XML that contains a dtd, and point it at the filesystem:

http://github.com/tenderlove/markup_validity/blob/master/lib/markup_validity/validator.rb#L11-32

I'm hesitant to use sub, only because we sometimes deal with very large files (in the hundred of MB range). I think I've nutted out the system XML catalog stuff, see http://github.com/yob/onix-dtd.>

Thanks again for your help.

adding the filter method on node set. closed by 90d4de3

Yo. This is fixed in master. Commit 1fefd59

I believe what ration meant to say was:

#!/usr/bin/env ruby

require 'nokogiri'

xml1 = "<test>
  <caption>Original caption</caption>
</test>"
xml2 = "<test>
  <caption>Replacement caption</caption>
</test>"
doc1 = Nokogiri::XML(xml1)
doc2 = Nokogiri::XML(xml2)
caption1 = doc1.xpath("//caption")[0]
caption2 = doc2.xpath("//caption")[0]
caption1.replace(caption2)

Node#replace reimplemented using reparent_node_with. Fixed by ebd4483

Ah yes, I didn't notice that the markdown broke the tags..

With this fix, doc2 will not contain the caption element. Is this the preferred behavior?
Personally I would assume the node is copied there, and something like "move" would actually move it.

yes, this is the preferred behavior. if you want to copy the node, copy it:

caption1.replace(caption2.dup)

Ok, thanks for the fix!

Moving was actually what I wanted, so this is perfect. I might not be the only one who doesn't automatically see this behavior, so maybe the documentation for replace deserves a comment about this side effect for the originating doc.

Commit bfa172d allows to set the RECOVER option for PushParser. This resolves the issue.

Ok, cool. I'll close this ticket then.

The main issue here is that FFI can't find the DLLs needed. I see that nokogiri does some tricks with PATH, but it won't work with dll loading, since LoadLibrary() call won't see PATH changes. So, either libs need to be specified with the full path, or they should be places in some place which is already on PATH, before JRuby starts.

Also it worth noting than on Jruby's master branch we changed a bit how ffi_lib works, so that it won't silently skip the DLL if it's not found, so with JRuby master version the failure is immediate and more clear. I also verified that if I put the DLLs into place on PATH and then start JRuby, nokogiri loads fine and works.

Here's the patch that solves the problem on Windows (under JRuby):
http://gist.github.com/233100

Essentially, since ffi_lib doesn't do any magic at all about the paths, we should provide the fully qualified path names for every DLL, and in Windows format, since those paths will be directly transferred to LoadLibrary() call.

I have tested this patch with Windows, JRuby 1.4.0 and Nokogiri 1.4.0 and it works for me.

ffi + windows + jruby dll loading fix (thanks, Vladimir Sizikov!). Closed by e4976fd

Thanks for the help on this.

fixed in 6510fe7

ah, for clarity, this is flavorjones. wrong login.

I concur. that was me.

closing. this will be in 1.4.1 sometime in the next week or so.

Node#matches? works in nodes contained by a DocumentFragment. closed by d41db1a

Fixed in 55fbf25

What version of libxml2 are you using? Past the contents of this command, if you can:

$ nokogiri -v

Also, if you could post the code to reproduce this, that would be great. So far, libxml2 is behaving how you want it to:

d = Nokogiri::XML('<root />')
d.root['foo'] = 'hello " world'

puts d.to_xml # => '<root foo="hello &quot; world"/>'

Closing as there has been no response.

バグレポートを送ってくれてありがとう。

修正しました！

Thanks for using Nokogiri.

Fixed in 7a77846

• ext/nokogiri/xml_document.c (Nokogiri_wrap_xml_document) init called after the tuple is set up. closed by 7fbf262

Closed by 597195f

NodeSet.wrap now preserves document structure. closed by f7388be.

and 2d3db36

This was merged in with fd6fc58

Right, but the :new_lines and :spacing options don't exist yet. I was meaning someone add code for these options so we can make the resulting xml completely compact (no indents, no new lines etc).

ah, oops. I thought this was the same thing.

The document on the builder is searchable. You can do:

builder.doc.at('//root')

In fact, Builder#doc just returns a Nokogiri::XML::Document that you can manipulate as your normally would.

Awesome. So all that's left is a way to remove new lines. At the moment we do

.gsub(/\n/, '')

But any content with new lines gets compacted too.

>> builder = Nokogiri::XML::Builder.new
>> builder.root { |xml| xml.test('hey') }
>> builder.doc.root.to_xml(:indent => 0)
=> "<root>\n<test>hey</test>\n</root>"

Is there a way to do this?

>> builder = Nokogiri::XML::Builder.new
>> builder.root { |xml| xml.test('hey') }
>> builder.doc.root.to_xml(:indent => 0, :new_lines => false)
=> "<root><test>hey</test></root>"

This is important for me. Because of my program is web robot that detects encoding and merge to utf before nokogiri work.

I think, nokogiri must ignore tag encoding if they set in initialize.

Unfortunately the encoding declared inside the document is king.

In 1.4.1 you'll be able to alias UTF-85 to UTF-8 like so:

Nokogiri::EncodingHandler.alias('UTF-8', 'UTF-85')

And that will let you parse the strange encoding declaration.

b631d67
38fdb08
a073613

BTW, I just discovered it also affects the OBJECT tag.

I don't think this is a bug. The default save options for to_xhtml say to format or "pretty print" the document. If your document contains space nodes, it will preserve them. If there are no blank nodes in the document, it will add them to make the output formatted.

If you don't want formatting, you can change the to_xhtml save options:

s = '<a><b>see</b></a>'
n = Nokogiri::HTML::DocumentFragment.parse(s)
puts n.to_xhtml(:save_with => Nokogiri::XML::Node::SaveOptions::AS_XHTML)

ah thanks for the tip. Might be nice to document these somewhere. I noticed libxml2 also doesn't document them.

duplicating erorrs works. yay! closed by 33922d7

What version of libxml2?

Looks like 2.7.3.

If you were on libxml2, 2.6.16, then I wouldn't be surprised. That version was very old an unstable.

I can't repro this (even with thousands of iterations), so I will assume it's a bug with 2.6.16. I am going to close this, but if you are able to repro with 2.7.3, please reopen this ticket. Thanks!

Shouldn't require thousands of iterations, it's a happens-every-time kind of bug. However, I may have been wrong about the version of nokogiri I was using. It might have been edge.

raising an exception if someone tries to reparent a *::Document. closed by c557764

Can you run "nokogiri -v" and add the output to this ticket?

It seems that that did the trick. I have not had any segfaults since.

libxml:
loaded: 2.7.3 binding: extension compiled: 2.7.3 nokogiri: 1.3.3

I am still busy testing it, but so far so good. I will keep you posted and close this ticket when I am sure that it is not an issue anymore.

Okay, sounds good. You might also want to try upgrading libxml2. The latest libxml2 is 2.7.5 and I know they've packed in a bunch of bug fixes. If that doesn't do the trick, would you mind sending us a sample of the XML you're using to make it crash? It shouldn't SEGV under any circumstances. :-)

:) I will do that. Thanks for the advice.

Any updates on this?

I'm closing this since there have been no updates. Please reopen if you're still having problems! Thanks!

Thanks for the heads up! http://ihighfive.com/

You are very whalecome: http://ihighfive.com/whale-high-five.php

have you added '/usr/local/lib' to your ld.so.conf file?

hello?

Closing. If you're still having troubles with this, please send an email to the mailing list:

http://groups.google.com/group/nokogiri-talk

I've applied the patch, but next time please make sure the tests pass. After applying the patch, I got these errors:

  1) Error:
test_raw_append(Nokogiri::XML::TestBuilder):
NoMethodError: undefined method `content' for nil:NilClass
    test/xml/test_builder.rb:127:in `test_raw_append'

  2) Error:
test_raw_append_with_instance_eval(Nokogiri::XML::TestBuilder):
NoMethodError: undefined method `content' for nil:NilClass
    test/xml/test_builder.rb:137:in `test_raw_append_with_instance_eval'

XML Builder can append raw strings. closed by 98b10d2

Yeah, I pasted the wrong patch. Sorry about that.

No problem! :-)

Works for me with libxml 2.7.3 and nokogiri 1.3.3

Still crashes for me using libxml2 2.7.5 and nokogiri 1.3.3 and ree-1.8.6-20090610. It does however work with normal MRI: ruby 1.8.6 (2009-08-04 patchlevel 383) [i686-darwin9.8.0]

This isn't crashing for me. I'm using:

[apatterson@higgins nokogiri (master)]$ ruby -v
ruby 1.8.6 (2008-08-11 patchlevel 287) [i686-darwin10.0.0]
Ruby Enterprise Edition 20090610
[apatterson@higgins nokogiri (master)]$ ruby -I lib bin/nokogiri -v
--- 
nokogiri: 1.3.3
warnings: []

libxml: 
  compiled: 2.7.5
  loaded: 2.7.5
  binding: extension
[apatterson@higgins nokogiri (master)]$

Can you try with the nightly? To install the nightly build, do this:

$ sudo gem install nokogiri -s http://tenderlovemaking.com

I can't reproduce this. Please reopen if the problem persists. I need more details to fix this if there is a problem. Thanks!

root cause:

"gcc -o conftest -I/usr/include/libxml2 -I/usr/include -I. -I/home/mike/builds/ruby-enterprise-1.8.6-20090610-install/lib/ruby/1.8/i686-linux -I/home/mike/code/nokogiri/ext/nokogiri -I/usr/include/libxml2 -I/usr/include -I. -I/home/mike/builds/ruby-enterprise-1.8.6-20090610-install/lib/ruby/1.8/i686-linux -I/home/mike/code/nokogiri/ext/nokogiri -g -O2  -g -DXP_UNIX -O3 -Wall -Wcast-qual -Wwrite-strings -Wconversion -Wmissing-noreturn -Winline conftest.c  -L/opt/local/lib -Wl,-R/opt/local/lib -L.  -rdynamic -Wl,-export-dynamic    -lexslt -lxslt -lxml2  -lruby-static -lexslt -lxslt -lxml2  -ldl -lcrypt -lm   -lc"
conftest.c: In function ‘t’:
conftest.c:3: warning: implicit declaration of function ‘xmlRelaxNGSetParserStructuredErrors’
/usr/bin/ld: cannot find -lruby-static
collect2: ld returned 1 exit status
checked program was:
/* begin */
1: /*top*/
2: int main() { return 0; }
3: int t() { xmlRelaxNGSetParserStructuredErrors(); return 0; }
/* end */

suggested fix:

diff --git a/ext/nokogiri/extconf.rb b/ext/nokogiri/extconf.rb
index 7c21e7d..b77552d 100644
--- a/ext/nokogiri/extconf.rb
+++ b/ext/nokogiri/extconf.rb
@@ -129,7 +129,7 @@ unless find_library('exslt', 'exsltFuncRegister', *LIB_DIRS)
   abort "libxslt is missing.  try 'port install libxslt' or 'yum install libxslt-devel'"
 end

-def nokogiri_link_command ldflags, opt='', libpath=$LIBPATH
+def nokogiri_link_command ldflags, opt='', libpath=$DEFLIBPATH|$LIBPATH
   old_link_command ldflags, opt, libpath
 end

proposed fix from Michael Reinsch:

- def nokogiri_link_command ldflags, opt='', libpath=$LIBPATH
+ def nokogiri_link_command ldflags, opt='', libpath=$DEFLIBPATH|$LIBPATH

which appears to work for me. Aaron, thoughts?

(which is what is used in mkmf.rb, link_command for ruby enterprise)

Ugh. This is a PITA, but basically we can't make everyone happy. If I add this patch, I may as well remove that "nokogiri_link_command" stuff all together. Let me try to explain why:

Someone has ruby installed in /usr/lib, they also have libxml2 installed in /usr/lib. They've installed a newer version of libxml2 in /usr/local. We try to be nice and search /opt/local/lib in addition to /usr/local/lib before falling back to /usr/lib. Unfortunately, if the custom directory (/opt/local or /usr/lib) isn't supplied to dir_config(), it won't search that path. We can only supply one directory. If mkmf doesn't find it in that directory, then it falls back to /usr/lib.

That means we either get /opt/local/lib or /usr/lib, unless the user intervenes with a --with-xml-lib=/whatever --with-xml-include=/whataver, or we use my Super Hack® code. Unfortunately my Super Hack® screws over people with custom ruby installs because it will never find the ruby-static library.

I'm going to apply this fix (and by apply, I mean remove my custom code). I'd rather it "just work" for people with custom ruby installs. People with custom libxml2 installs can use the command line arguments.

removing my Super Hack® closed by fbe7217

Aaron, the --with-xml-lib and --with-xml-include options do not appear to affect have_func(), since it consistently uses the wrong header files.

will look at this ASAP.

i've got a repro case and a possible fix.

Can you try out the nightly build? I believe this may already be fixed.

To get the nightly, do this:

$ sudo gem install nokogiri -s http://tenderlovemaking.com

I tested it against nokogiri/master and the nightly, the crash still happens.

But I increased the 'input' length in the test script, the crash should now happen more often (almost every time now on three of my machines).

git://github.com/retoo/nokogiri-bug.git

Thanks!

retoo - the fix tenderlove is referring to does not apply to your/my situation, sorry about that. We have a repro case and a potential fix, and we are working on it. Thanks!

@retoo I think we've got a fix in. Can you pull the latest nightly and see if that fixes your crash?

It doesn't crash anymore here. Bisect says that it has been fixed in c753c8d.

Thank you guys!

Reto

Closing...

Hi guys,

I might have the same issue, I get the following error message while downloading a an xml file and opening it using nokogiri:

res = Net::HTTP.post_form(URI.parse(....), {...})
doc = Nokogiri::XML(Nokogiri::XML(res.body).xpath("//text()").to_s.gsub("& lt;", "<").gsub("& gt;", ">"))

I have installed the latest nightly on OS X 10.5.6.

/Library/Ruby/Gems/1.8/gems/nokogiri-1.3.3.20091004000018/lib/nokogiri/xml/document.rb:33: [BUG] Segmentation fault ruby 1.8.7 (2008-08-11 patchlevel 72) [universal-darwin10.0]

Abort trap

Regards
William

Hmm, probably not, 1. It didn't crash on os x in my case, 2. it crashed when I tried to replace() some parts of it, not during any xpath/constructor calls.

Nonetheless, I would report this (perhaps in a new, unused fresh ticket :D). Does this crash everytime? If yes, seperate the two constructor calls like:

step 1 = XML(input).xpath.to_s.gub
puts "I survived so far
step2 = XML(step1)
puts "should never reach this point"

It would be very helpful if you could give the output of res.body. Or even better, the minimum of res.body which still triggers the crash.

Cheers.

Hey Reto, it errors out on

doc = Nokogiri::XML(res.body).xpath("//text()").to_s.gsub("&lt ;", "<").gsub("&gt ;", ">")

I have about 10 different XML files, and it crashes randomly on the different files, so I can't say that it's one specific file.

The XML files vary in size from 3mb to 150mb.

The files are very basic XML:

<?xml version="1.0" encoding="utf-8"?>
<string>....</string>

where the string element contains escaped XML. Unfortunately the XML files are data we receive from an external vendor, so not really anything we can do about that.

I can try to normalize the data using gsub and then use nokogiri on the xmlified data.

Hi, can you please open a new issue for this? It appears to be unrelated to this actual issue, which has been solved in the nightlies.

Ok, I'll do that.

Created a new ticket: http://github.com/tenderlove/nokogiri/issues/#issue/144

i close this one again. thx for the help guys!

Closing this because we fixed it. :-)

We are having the same problems....

Same problems here which is unfortunate since the community really needs a cross-platform XML library. The developers of our project work on Windows, Linux as well as Solaris and deploy on Solaris. Nokogiri would be perfect in this regard if this bug was just fixed.

Please, this problem has existed for about a year and is a show-stopper for many projects. I know the developers time are limited and they do this in their spare time, but if wide usage of the library is of any interest, addressing this bug is a quick win to expand the use of Nokogiri.

Hey everyone, this is a bug in JRuby. I've filed a ticket with them here:

http://jira.codehaus.org/browse/JRUBY-4052

Once they get it sorted out, I will close this ticket. :-)

I am confused about what it is we're not doing. If calloc is a (fairly) standard POSIX function, what is it we should be doing differently?

@headius I think the libc functions should be loaded by default. Wayne gives a workaround in the JIRA ticket, but I think it's unreasonable to require me to change my FFI code depending on the platform. The current code works on everything but windows.

Fix added to git://github.com/jojje/nokogiri.git for people to try having access to a Windows environment. Requires JRuby 1.4.0RC1 or higher due to some needed fixes regarding FFI.

I applied this:

0555c69

I'm closing this ticket since it should be fixed for JRuby / Windows people with 1.4.0

Strange. Are you sure nokogiri -v returns the same thing on the linux box? That seems crazy.

Yeah - I thought I was going crazy so I even did a diff. Weirdest thing I've seen... AFAIK libxml2 doesn't really depend on much does it? Or is there something else that Nokogiri depends on that might be causing this? I tried libxml2 2.7.2 just in case, but still had the same problem.

libxml2 only depends on iconv and zlib. Neither of those should cause this problem.

What linux are you running?

Gentoo (default/linux/x86/2008.0 profile) over the 2.6.18-xenU-ec2-v1.0 kernel

libxml2: 2.7.3-r2
libc: 2.8_p20080602-r1
zlib: 1.2.3-r1

Okay. I'll get a gentoo box up and running. Might be a little while before I get this one to repro. :-(

Thanks a ton! In the meantime I'm trying to upgrade glibc and anything else that might be out of date, just to try some different versions of things.

FWIW - The new glibc (2.9_p20081201-r2) didn't affect anything.

Here's another take on it, if it helps at all:

require 'test/unit'
require 'rubygems'
require 'nokogiri'

module Nokogiri::XML
  class Node
    include Test::Unit::Assertions

    def inner_html=(tags)
      children.each { |x| x.remove }

      assert_equal ['sup', 'sub'], document.fragment(tags).children.map {|n| n.name }
      document.fragment(tags).children.to_a.each do |node|
        add_child node
      end
      self
    end
  end
end

class BugTest < Test::Unit::TestCase
    def test_should_parse_inner_html
        text = '<base><one>1</one><two>2</two></base>'

        doc = Nokogiri::XML(text)
        base_tag = doc.at('base')
        base_tag.inner_html = "<sup>#{base_tag.at('one').inner_text}</sup><sub>#{base_tag.at('two').inner_text}</sub>"

        assert_equal ['sup', 'sub'], base_tag.children.map {|n| n.name }
    end
end

Successful return on the MacPorts install, and on the Linux install:

  1) Failure:
test_should_parse_inner_html(BugTest)
    [oo2.rb:12:in `inner_html='
     oo2.rb:27:in `test_should_parse_inner_html']:
<["sup", "sub"]> expected but was
<["sup"]>.

In fact, even just this code returns only the first element and not the other:

Nokogiri::XML::DocumentFragment.parse("<one>1</one><two>2</two>")

Unless it's wrapped in another outer element like:

<x><one>1</one><two>2</two></x>

...then it returns the whole thing. And then obviously on the MacPorts install it returns an accurate copy regardless of the surrounding element.

I think I narrowed this down finally. For whatever reason, my local copy of Nokogiri (even though the gem was labeled 1.3.3) showed this diff from the copy on the linux machine (which was recently installed):

8,9c8,13
<         @html_eh        = node.kind_of? Nokogiri::HTML::DocumentFragment
< 
---
>         @klass          = if node.kind_of?(Nokogiri::HTML::DocumentFragment)
>                             Nokogiri::HTML::DocumentFragment
>                           else
>                             Nokogiri::XML::DocumentFragment
>                           end
>         #
23,25c27,28
<         regex = @html_eh ? %r{^\s*<#{Regexp.escape(name)}}i :
<                            %r{^\s*<#{Regexp.escape(name)}}
< 
---
>         regex = (@klass == Nokogiri::HTML::DocumentFragment) ? %r{^\s*<#{Regexp.escape(name)}}i \
>                                                              : %r{^\s*<#{Regexp.escape(name)}}

So a fresh install on my MacPorts version now fails as well - lol - not quite the expected result. However, installing the gem from the master works great - so looks like this was already fixed ;)

Ugh. You're right. It fails against 1.3.3. I was checking against master. :-(

I guess I can stop fighting with VirtualBox now. Thanks for letting me know!