NWN2: Library and tool to fix NWN2 XML files

[rjshae]

This pull provides the class XMLFix, which can accept a
SeekableReadStream containing the contents of a NWN2 XML file and
return a SeekableReadStream with the XML in a valid form. This can
be exercised with the included fix2xml.exe build, which reads in a
NWN2 XML file and outputs a corrected XML file. The latter tool was
used to bulk test the stock NWN2 XML files and the XML files from
some NWN2 mods. The result was processed with a call to
'xmllint --noout filename.xml' to check for valid XML. All of the
converted test files passed validation.

The mods used to test the class were:

Tchos' HD UI panels and dialogue compilation and expansion
SlimGUI
Black UI
Kaldor Silverwand Context Menu Additions
I manually checked a sample of the output files looking for
irregularities, and these have been corrected.

The code required the addition of an at() call to the UString class
and a modification of the UString::trim* functions to check for
whitespace characters rather than just ' '.

Finally, it includes a unit test for producing valid XML output.

[DrMcCoy]

From the outside, that looks far better, yes :)

I can look at it in more depth in the evening

[DrMcCoy]

Yup, this PR is now far more what I was looking for. Thanks! :)

There's just a few style things to fix, and the line-ending thing (i.e. you've commited everything with Windows line endings, while we use Unix line endings).

And another thing:

The only state the XMLFixer class keeps around is the button counter, right? Would it be possible to move that counter completely into the function scope, and just pass it around as a parameter to isBadUIButtonRange()?

That way, you could make fixXMLStream() completely static, and the fixer wouldn't need an XMLFixer instance at all. You'd just call XMLFixer::fixXMLStream() and be done.

[DrMcCoy]

Yup, this PR is now far more what I was looking for. Thanks! :)

There's just a few style things to fix, and the line-ending thing (i.e. you've commited everything with Windows line endings, while we use Unix line endings).

And another thing:

The only state the XMLFixer class keeps around is the button counter, right? Would it be possible to move that counter completely into the function scope, and just pass it around as a parameter to isBadUIButtonRange()?

That way, you could make fixXMLStream() completely static, and the fixer wouldn't need an XMLFixer instance at all. You'd just call XMLFixer::fixXMLStream() and be done.

[rjshae]

Thanks. I've tried to address your concerns and I added a man page.

[DrMcCoy]

Yeah, that looks better now :)

However, "Common:" in the commit message of 4dc5666 should be "COMMON:".

Also, why did you name the tool fix2xml? Shouldn't it be rather fixnwn2xml?

Also also, I'd rather have the default output of the tool, when no output name is given, be stdout instead of a file with _Fixed appended. We have a class, StdoutStream, for that.

[DrMcCoy]

Oh, and that reminds me: it would be nice if the tool could read from stdin as well. If no input file name is given, for example.

We do have a StdinStream class for that... However, you can't, of course, seek the stdin, so StdinStream inherits from ReadStream, not SeekableReadStream.

Skimming over the code, you don't necessarily need to seek, right? The only thing you call is Common::readStringLine(), which does require a SeekableReadStream. But looking at that code, it and readFakeChar() don't really need to seek either, so they could be dropped to use ReadStream.

[DrMcCoy]

Though, that might be something for another PR.

Ignore that last comment of mine for now, only focus on the one before :)

[rjshae]

Everything's done except the write to stdout. I'm a little stumped there:

void convert(Common::UString &inFile, Common::UString &outFile) {
	// Read the input file into memory
	Common::ScopedPtr<Common::SeekableReadStream> in(Common::ReadFile::readIntoMemory(inFile));
	Common::ScopedPtr<Common::WriteStream> out(openFileOrStdOut(outFile));

	// Filter the input
	Common::SeekableReadStream *fixed = Aurora::XMLFixer::fixXMLStream(*in);

	// Write to output
	out.writeStream(fixed);
	out.flush();
	out.close();

	if (!outFile.empty())
		status("Converted \"%s\" to \"%s\"", inFile.c_str(), outFile.c_str());
}

Gives:

$ make
make  all-am
make[1]: Entering directory '/home/rjhall/xoreos-tools'
  CXX      src/fixnwn2xml.o
src/fixnwn2xml.cpp: In function 'void convert(Common::UString&, Common::UString&)':
src/fixnwn2xml.cpp:101:6: error: 'class Common::ScopedPtr<Common::WriteStream>' has no member named 'writeStream'
  out.writeStream(fixed);
      ^~~~~~~~~~~
src/fixnwn2xml.cpp:102:6: error: 'class Common::ScopedPtr<Common::WriteStream>' has no member named 'flush'
  out.flush();
      ^~~~~
src/fixnwn2xml.cpp:103:6: error: 'class Common::ScopedPtr<Common::WriteStream>' has no member named 'close'
  out.close();
      ^~~~~
make[1]: *** [Makefile:5042: src/fixnwn2xml.o] Error 1
make[1]: Leaving directory '/home/rjhall/xoreos-tools'
make: *** [Makefile:3047: all] Error 2

Even though I see it in the WriteStream call in the WriteStream header. Ah well, I'll take a look later. Other projects call.

[DrMcCoy]

The ScopedPtr has no writeStream. The containing WriteStream has. So you need to use out->writeStream()

[rjshae]

Hello DcMcCoy,

I believe I've addressed your concerns. Is there some other hold up? Thanks.

[rjshae]

Hello Sven, I've updated the code according to your requested changes, but there's been no activity since that time. I'm not sure if the notification system is working properly. May I request a review at your earliest convenience? Thank you kindly. Ed.: Hmm, this appears to be a closed loop system. Ah well.

…

On Sun, Sep 30, 2018 at 12:20 PM Sven Hesse ***@***.***> wrote: The ScopedPtr has no writeStream. The containing WriteStream has. So you need to use out->writeStream() — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#24 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ApFfrAoh-NTYBCkAKC7cA3YGmlBiKHfoks5ugRmZgaJpZM4W1-mO> .

-- Bob "A fish rots from the head down" -- Ancient proverb

[DrMcCoy]

Yes, I'm sorry, I just didn't have enough time to properly look at your PR. The hold-up is me.

From a cursory look, it really seems to be great now (thanks again!), but I still want to go through all the commits one by one.

I see if I can do that tomorrow evening, and possibly merge this PR then too.

[rjshae]

Ah, okay. No problem then; whenever you can get to it. Thank you.

[rjshae]

Ah, okay. No problem then; whenever you can get to it. Thank you.

…

On Sun, Oct 21, 2018 at 2:47 PM Sven Hesse ***@***.***> wrote: Yes, I'm sorry, I just didn't have enough time to properly look at your PR. The hold-up is me. From a cursory look, it really seems to be great now (thanks again!), but I still want to go through all the commits one by one. I see if I can do that tomorrow evening, and possibly merge this PR then too. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#24 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ApFfrCcePocWwRlW-hPbC7ZEbFnKF-A6ks5unOtpgaJpZM4W1-mO> .

-- Bob "A fish rots from the head down" -- Ancient proverb

[DrMcCoy]

Yeah, this looks pretty good now, thanks :)

How do you want to be credited in the AUTHORS file? The rough format I've been using is Name (Nickname) <email adress>, though all fields are basically optional.

Since you do have your name (and email address as well, of course) in the commits, I'd use Bob Hall (rjshae) and said email address, unless that's not what you want.

[rjshae]

I'm fine with your proposed credit format. Thanks for taking a look.

…

On Sun, Oct 28, 2018 at 3:31 AM Sven Hesse ***@***.***> wrote: Yeah, this looks pretty good now, thanks :) How do you want to be credited in the AUTHORS file? The rough format I've been using is Name (Nickname) <email adress>, though all fields are basically optional. Since you do have your name (and email address as well, of course) in the commits, I'd use Bob Hall (rjshae) and said email address, unless that's not what you want. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#24 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ApFfrBayffqIfgnhq08_HezS5Guaq4IKks5upYdjgaJpZM4W1-mO> .

-- Bob "A fish rots from the head down" -- Ancient proverb

[DrMcCoy]

Okay, merged as 41aaf31...d641a56, thanks! :)