Implement XML parsing using stdlib

[HarshNarayanJha]

This PR replaces github.com/clbanning/mxj/v2 and uses encoding/xml xml.Decoder to parse xml and extract urls within.

EDIT: All tests pass now. The PR is complete

Co-Author: @yzqzss

Only two tests fail, I am trying to fix those

Tests

go: downloading git.archive.org/wb/gocrawlhq v1.2.13
internal/pkg/crawl/config.go:11:2: unrecognized import path "git.archive.org/wb/gocrawlhq": https fetch: Get "https://git.archive.org/wb/gocrawlhq?go-get=1": dial tcp 207.241.235.124:443: i/o timeout
=== RUN   TestJSON
=== RUN   TestJSON/Valid_JSON_with_URLs
=== RUN   TestJSON/Invalid_JSON
=== RUN   TestJSON/JSON_with_no_URLs
=== RUN   TestJSON/JSON_with_URLs_in_various_fields
=== RUN   TestJSON/JSON_with_array_of_URLs
--- PASS: TestJSON (0.00s)
    --- PASS: TestJSON/Valid_JSON_with_URLs (0.00s)
    --- PASS: TestJSON/Invalid_JSON (0.00s)
    --- PASS: TestJSON/JSON_with_no_URLs (0.00s)
    --- PASS: TestJSON/JSON_with_URLs_in_various_fields (0.00s)
    --- PASS: TestJSON/JSON_with_array_of_URLs (0.00s)
=== RUN   TestXML
=== RUN   TestXML/Valid_XML_with_URLs
=== RUN   TestXML/Empty_XML
=== RUN   TestXML/Invalid_XML
=== RUN   TestXML/XML_with_invalid_URL
=== RUN   TestXML/Huge_sitemap
    xml_test.go:88: XML() gotURLs count = 10000, want 100002
--- FAIL: TestXML (0.76s)
    --- PASS: TestXML/Valid_XML_with_URLs (0.00s)
    --- PASS: TestXML/Empty_XML (0.00s)
    --- PASS: TestXML/Invalid_XML (0.00s)
    --- PASS: TestXML/XML_with_invalid_URL (0.00s)
    --- FAIL: TestXML/Huge_sitemap (0.66s)
=== RUN   TestXMLBodyReadError
    xml_test.go:127: XML() expected error, got nil
--- FAIL: TestXMLBodyReadError (0.00s)
FAIL
FAIL	github.com/internetarchive/Zeno/internal/pkg/crawl/extractor	0.780s
FAIL

Closes #84

[CorentinB]

Hey, thank you, how is it going?

[HarshNarayanJha]

Not going well. I am not sure what am I failing to capture in that big XML file. Any ideas @CorentinB ?

[yzqzss]

		case xml.StartElement:
			startElement = tok
			currentNode = &LeafNode{Path: startElement.Name.Local}
			for _, attr := range tok.Attr {
				if strings.HasPrefix(attr.Value, "http") {
					parsedURL, err := url.Parse(attr.Value)
					if err == nil {
						URLs = append(URLs, parsedURL)
					}
				}
			}

this fixed the Huge sitemap test by extracting XML attributes.
Now, the URLs' size and content match the previous tests. :)

[HarshNarayanJha]

Right, opening tags also have urls in some cases.

Now the only left is TestXMLBodyReadError, gotta look at it

[yzqzss]

I think the TestXMLBodyReadError test is invalid(?) since NopCloser certainly won't return an EOF error on xmlBody, err := io.ReadAll(resp.Body)

[HarshNarayanJha]

Possible, but they did pass previously. Sure, it does not return an EOF on read. For the test to pass, I had to decode the Token once and catch the error. (and seek back for the loop)

_, err = decoder.Token()
if err != nil {
	return nil, sitemap, err
}

// seek back to 0 if we are still here
reader.Seek(0, 0)
decoder = xml.NewDecoder(reader)

Catching this in the loop won't work cleanly, since I want to know if EOF was somewhere in-between the file (invalid XML), or at the start (this error)

I will push these changes

[yzqzss]

LGTM :)

[CorentinB]

Thanks guys!

[HarshNarayanJha]

You're welcome