Incorrectly changing — to - (emdash [alt 0151]) to hypens.

[JustAGuyTryingToCodeSomething]

When cleaning up an MSWord document,

<p class=CSP-ChapterBodyText><span lang=EN-US style='font-size:14.0pt'>“I thought—”</span></p>

is converted to

	<p>
	"I thought-"
	</p>

_If you look closes the — emdash in the original text has been changed to a hyphen, which is actually introducing a grammatical error into the text, the original emdash should be persisted. _

I've included my config file and how I run the tool below. I'm not sure how to check, but I'm 99% sure this is version 5.6.0.

Note: The change of the quote characters is interesting, and I'd be interested to know why that happens, but I'm not reporting that as an issue here.

`.\tidy.exe -config .\htmltidy.config -o .\tidied.html .\original.html

add-xml-space:               no
add-meta-charset:            no
anchor-as-name:              yes
ascii-chars:                 no
assume-xml-procins:          no
bare:                        yes
break-before-br:             no
char-encoding:               utf8
clean:                       no
coerce-endtags:              yes
add-xml-decl:                no
css-prefix:                  c
custom-tags:                 no
decorate-inferred-ul:        no
doctype:                     auto
drop-empty-elements:         yes
drop-empty-paras:            yes
drop-proprietary-attributes: no
enclose-block-text:          no
enclose-text:                no
escape-cdata:                no
fix-backslash:               yes
escape-scripts:              yes
fix-bad-comments:            no
fix-style-tags:              yes
fix-uri:                     yes
force-output:                no
gdoc:                        no
gnu-emacs:                   no
hide-comments:               yes
indent:                      yes
indent-attributes:           no
indent-cdata:                no
indent-spaces:               4
indent-with-tabs:            yes
input-encoding:              utf8
input-xml:                   no
join-classes:                no
keep-tabs:                   no
keep-time:                   no
literal-attributes:          no
join-styles:                 yes
logical-emphasis:            no
lower-literals:              yes
markup:                      yes
merge-divs:                  auto
merge-emphasis:              yes
merge-spans:                 auto
mute-id:                     no
ncr:                         yes
new-blocklevel-tags:         no
omit-optional-tags:          no
output-bom:                  auto
output-encoding:             utf8
output-html:                 no
output-xhtml:                no
output-xml:                  no
preserve-entities:           no
punctuation-wrap:            no
quiet:                       no
quote-ampersand:             yes
quote-marks:                 no
quote-nbsp:                  yes
repeated-attributes:         keep-last
replace-color:               no
show-body-only:              no
show-errors:                 6
show-info:                   yes
show-meta-change:            no
show-warnings:               yes
skip-nested:                 yes
sort-attributes:             none
strict-tags-attributes:      no
tab-size:                    8
tidy-mark:                   yes
uppercase-attributes:        no
uppercase-tags:              no
vertical-space:              no
warn-proprietary-attributes: yes
word-2000:                   yes
wrap:                        68
wrap-asp:                    yes
wrap-attributes:             no
wrap-jste:                   yes
wrap-php:                    yes
write-back:                  no
wrap-script-literals:        no
wrap-sections:               yes

[geoffmcl]

@JustAGuyTryingToCodeSomething, yes using your config, the emdash is indeed changed to a hyphen...

However, if I remove that config, there is no problem, it is output as is...

I am afraid I do not have the time to review your giant config list... maybe someone else can spot the likely option, or options, that changes, causes, this action...

Please delete your config, and start with the minimum, and try again... track the problem down... 1 by 1...

This might also address the quotes change as well...

Look forward to further feedback, discussion, what worked, what didn't, etc... thanks...

[JustAGuyCoding]

@geoffmcl It's been a while since I tried this. My notes say it's the "bare" option which is responsible. Not sure why I didn't say so in the bug report. I haven't retested but will sometime this week and get back to you. Thanks for looking in to this.

[JustAGuyCoding]

OK. Quickly re-ran the test. When using a config file with only bare:no the output is correct. When using a config file with only bare:yes the output is incorrect.

The notes from http://tidy.sourceforge.net/docs/quickref.html#bare say:

bare | Top
Type: BooleanDefault: noExample: y/n, yes/no, t/f, true/false, 1/0 |  
This option specifies if Tidy should strip Microsoft specific HTML from Word 2000 documents, and output spaces rather than non-breaking spaces where they exist in the input.

That makes sense, as the option was being used to clean an MSWord file.

[geoffmcl]

@JustAGuyCoding, wow, your link is truly ancient, circa 2008 ;=))

But the info bare remains unchanged...

I guess converting a emdash to hyphen is another example of strip Microsoft specific, or convert...

The unix man tidy is a bit more forthright about -b use == strip out smart quotes and em dashes, etc.... so it is not just MSWord files that are effected...

Maybe there is a small case here to update the docs? Or not...

Is there anything still outstanding here? Can this be closed? Thanks...

[JustAGuyCoding]

Sorry about the link!

Agreed, I see it says the processing directive -b strips smart quotes and em dashes. I guess that makes it expected behaviour and not a bug. Although it's converting\replacing them not stripping them out.

I think a footnote in the docs would certainly be helpful for future users as it's not obvious behaviour. It seems odd that the same flag is used for both 1) stripping smart quotes and em dashes (and etc - what else exactly?!) and 2) MSWord specific HTML. It's not obvious to me why the two go together.

My use case for this is to automate MSWord document to clean HTML for Kindle book publishing. I'm told by authors\editors that substituting a hyphen for an em-dash and visa versa is a grammatical error; I suspect the use of straight or smart quotes is more a style issue. The obvious issue for me being that I can't clean an MSWord document without using the bare=yes option but that I need the copy to remain unchanged.

Happy for you to close this if you don't think it's something which should be pursued.

Thanks for all your help.

[geoffmcl]

@JustAGuyCoding, documentation aside, why do you think you need to have bare on?

It seems to be doing things you do not want, desire...

I just did a quick scan through the code, trying to determine exactly what bare does, when switched on... its default is off...

Some code snippets - tidylib.c

int tidyDocSaveStream( TidyDocImpl* doc, StreamOut* out )
...
    Bool makeBare     = cfgBool(doc, TidyMakeBare);
...
    if ((makeClean && asciiChars) || makeBare)
        TY_(DowngradeTypography)(doc, &doc->root);

    if (makeBare)
        /* Note: no longer replaces   in */
        /* attribute values / non-text tokens */
        TY_(NormalizeSpaces)(doc->lexer, &doc->root);
    else
        TY_(ReplacePreformattedSpaces)(doc, &doc->root);

So with bare, the DowngradeTypography service deals with replacing, emdash and quotes, to wit -

                    switch (c)
                    {
                    case 0x2013: /* en dash */
                    case 0x2014: /* em dash */
                        c = '-';
                        break;
                    case 0x2018: /* left single  quotation mark */
                    case 0x2019: /* right single quotation mark */
                    case 0x201A: /* single low-9 quotation mark */
                        c = '\'';
                        break;
                    case 0x201C: /* left double  quotation mark */
                    case 0x201D: /* right double quotation mark */
                    case 0x201E: /* double low-9 quotation mark */
                        c = '"';
                        break;
                    }

And with bare all non-breaking space (nbs), code 160, in NormalizeSpaces, converted to space, while normally only nbs in <pre> tags is converted...

The bare also appears in the word-2000 service, but only if it does not appear to be a MSWord doc... forcing xmlns deletion on a non-word-2000 doc...

void TY_(CleanWord2000)( TidyDocImpl* doc, Node *node)
...
    while ( node ) {
        if ( nodeIsHTML(node) ) { /* get rid of Word's xmlns attributes */
            /* check that it's a Word 2000 document */
            if ( !TY_(GetAttrByName)(node, "xmlns:o") &&
                 !cfgBool(doc, TidyMakeBare) )
                return;

In other words, if you are passing valid Word 2000 document, what are you missing if you leave bare as its default NO?

If you think there should be a documentation improvement, enhancement, change, ... then that could be a new issue, hopefully with patches, or PR... ie some wording changes to be discussed... our docs always need reviewing... thanks...

Meantime look forward to your feedback, tests, ... with option --bare off, given after the config file read... or delete it... some diffs to showing the change... if you think you are still missing, something... thanks...

[JustAGuyCoding]

Okay, I can offer some clarity (numbered because it helps me think and type at the same time!):

1. The MSWord documents I export are using the Save As > Web Page(Filtered) option, which seems to be the best option to limit the amount of HTML crud MSWord exports. These files do not have the xmlns:o attribute set on the HTML tag.

2. On reflection HTMLTidy's word-2000=yes setting is targeting MSWord documents export as Save As > Web Page. These do have xmlns:o set.

3. The blend of HTMLTIdy configuration settings I've been using to clean the documents is essentially

bare:                        yes
char-encoding:        utf8
output-encoding:    utf8
word-2000:              yes

with everything else set to the default values.

4. Reviewing the output for different config settings the results seem to be:
   4.1

bare:                        no
char-encoding:        utf8
output-encoding:    utf8
word-2000:              no

Leaves in <span lang="EN-US" style='font-size:14.0pt'>“ and some arbitrary classes <p class="CSP-ChapterBodyText">Her smile became a grimace. "Oh."</p> which MSWord inserts [I think it's asking a bit too much of HTMLTidy to clean these up]. The em-dashes and quotes remain unchanged.

4.2

bare:                        yes
char-encoding:        utf8
output-encoding:    utf8
word-2000:              yes

Removes the <span lang="EN-US" style='font-size:14.0pt'>“ but leaves the arbitrary classes, which is better.
Filtered and Non-filtered files have the em-dashes and quotes changed (expected because bare=yes).
Non-filtered files have more header junk but that's expected.

4.3

bare:                        no
char-encoding:        utf8
output-encoding:    utf8
word-2000:              yes

Non-filtered files look good. Spans are removed, arbitrary classes remain, em-dashes and quotes are not replaced. This is perhaps in a good enough state for me to use given a little manual clean up.
Filtered as expected is not cleaned up.

5. Why use bare and word-2000 together? A very good question.

I suspect I found the result in 4.3 wasn't cleaning up the Filtered file, so went back to the documents and found that bare was also available to clean up MSWord files so I tried that, and hey presto it worked (I didn't spot the em-dashes and quote side effect).

To be fair the documentation says of word-2000 "strip out all the surplus stuff Microsoft Word 2000 inserts when you save Word documents as "Web pages". " so I shouldn't have expected it to clean filtered files.

6. Where does that leave us?

• I'm going to export MSWord files to Non-filtered Web Pages on my next run - with bare=no - as this seems the best option for my use case.

• The description of the bare config option could be tightened up (https://api.html-tidy.org/tidy/quickref_5.6.0.html#bare). I would suggest to a) indicate it's use as alternative to word-2000 for use of Filtered MSWord Web Page exports and highlight it's effect on em-dashes and quotes. I would offer to create a new issue and provide pull request but I'm hesitant because I'm not sure what the bare option is really trying to achieve. I don't understand why it converts the dashes or quotes or what it's use case is.

Sorry all the above is sooo long. Hopefully that provides some clarity and I haven't mixed up my test results (the results make sense to me).

I think you can close this issue. The question is whether to open another one on the purpose and documentation of the bare option!

Thanks for working through this with me.

[geoffmcl]

@JustAGuyCoding thank you for doing the tests... the results seem to make sense... given what I know...

And I think you could further shrink your config, by removing the 2 encodings, since 'utf8' is the default, in both cases...

Unfortunately, the commit history of the introduction of the bare and word-2000 options is hidden in the old sf.net CVS repository, Tidy's former home, pre circa 2009... and I do not know how to extract it... we used to be able to browse, review, but that seems missing now...

On bare, I found an old bug report, 2011-11-28, which suggests the docs are misleading, and I agree 110% with that...

Exactly!, what is bares use case? Is there a case to replace emdash, quotes, nbsp with ASCII only?

And what has this got to do with word-2000, other than that is a sample place where they are used?

So, to begin to bring the documentation into line I suggest something like -

diff --git a/include/tidyenum.h b/include/tidyenum.h
index 3daee5b..b2d4d06 100644
--- a/include/tidyenum.h
+++ b/include/tidyenum.h
@@ -610,7 +610,7 @@ typedef enum
     TidyLiteralAttribs,          /**< If true attributes may use newlines */
     TidyLogicalEmphasis,         /**< Replace i by em and b by strong */
     TidyLowerLiterals,           /**< Folds known attribute values to lower case */
-    TidyMakeBare,                /**< Make bare HTML: remove Microsoft cruft */
+    TidyMakeBare,                /**< Replace smart quotes and em dashes with ASCII. */
     TidyMakeClean,               /**< Replace presentational clutter by style rules */
     TidyMark,                    /**< Add meta element indicating tidied doc */
     TidyMergeDivs,               /**< Merge multiple DIVs */
diff --git a/src/language_en.h b/src/language_en.h
index 60bde02..eab5567 100644
--- a/src/language_en.h
+++ b/src/language_en.h
@@ -786,9 +786,9 @@ static languageDefinition language_en = { whichPluralForm_en, {
       - The strings "Tidy" and "HTML Tidy" are the program name and must not
       be translated. */
         TidyMakeBare,                 0,
-        "This option specifies if Tidy should strip Microsoft specific HTML "
-        "from Word 2000 documents, and output spaces rather than non-breaking "
-        "spaces where they exist in the input. "
+        "This option specifies if Tidy should replace smart quotes and em dashes with "
+        "ASCII, and output spaces rather than non-breaking "
+        "spaces, where they exist in the input. "
     },
     {/* Important notes for translators:
       - Use only <code></code>, <var></var>, <em></em>, <strong></strong>, and

That, at least says exactly what it is coded to do... what do you, others, think?

It remove the misleading that this is only for MSWord docs...

If I read you correctly, Word export filtered option, does NOT include <html xmlns:o="...">, the signature tidy seeks...

Ok, that's interesting, but does tidy still clean out all of what you want? Seems not...

So if not, why not just use the non-filtered export? Does this get you what you want? You seem to indicate yes...

In other words, with word-2000, is there any use case where bear must also be on?

If yes, where, how, to document this option dependency more clearly?

What other docs changes would you suggest? Or we could move this to a new issue...

Look forward to further feedback, comments, patches, PR, etc... or a new issue... thanks...

[JustAGuyCoding]

Everything you say seems true and right to me.

• I think it's time to create a new issue for "Bare: What's the use case?" and link to this discussion. I'll kick start that over the weekend for you and we'll see where it goes.

• The word-2000 option works best for me, in terms of using HTMLTidy to clean the cruft out of an MSWord export, but only when used against a Web Page export (containing xmlns:o), as opposed to a Web Page (Filtered export).

• Web Page (Filtered) is the best, most often recommended, way to export MSWord documents with the least amount of MS specific\custom cruft(attributes, classes, tags). I haven't found a third party plugin alternative to export HTML (back in the day you could write VBA code to do this kind of thing). So, that's why I used that export option and why I suspect most others will go the same route; The HTMLTidy documents even seem to suggest it over using word-2000.

-It should probably be noted that I'm exporting from MSWord Microsoft 365 version in 2020 (can't find a definite version reference), and there have to have been modifications to the export format since word-2000. Personally I think HTMLTidy should be fairly relaxed about this, fixing issues where and when they're reported.

Once again, thanks for all the help and discussion.

[JustAGuyCoding]

Please feel free to close this issue :)

[geoffmcl]

@JustAGuyCoding yeah, by all means open a new bare issue... thanks...

But I feel I am still missing something... maybe, what role, if any, does the option bare have in the word-2000 option?

Maybe you have answered that, but I just can not see it yet...

Yes, I am questioning the suggestion, You should consider ... Filtered. in our docs. Should that be removed, modified, clarified, in any way?

Only you as a word-2000 user, can make that judgment call... from your reports, it seems it could be at least qualified, a little, by say But is has been reported the non-filtered export is best for Tidy use.... T/F??? ... or something else...

I came across another MS Word open issue #855... there could be others...

I am open to fixing issues where and when they're reported, if it can be nailed down, spec'ed, with sample html, maybe finding the code that does this, patches, pseudo code, code samples, maybe even a PR... I need code help...

Look forward to further feedback, comments, patches, PR, etc... or a new issue... thanks...

[JustAGuyCoding]

But I feel I am still missing something... maybe, what role, if any, does the option bare have in the word-2000 option?

I suspect you're supposed to use one option or the other to target either the Web Page (filtered) or Web-Page (non-filtered) MSWord HTML exports respectively.

You note above that bare will have an effect on Web Page (filtered) inputs IF word-2000 and bare are both set - it seems like someone is trying to fix the config mistake, working on the logic that if someone supplies a Web Page (filtered) document with bare and word-2000 set they probably just want to run bare.

Yes, I am questioning the suggestion, You should consider ... Filtered. in our docs. Should that be removed, modified, clarified, in any way?
Probably not. I think the wording should be tweaked to say word-2000 specifically targets Web Page (non-filtered) exports. The bare wording should probably be tweaked to say it targets Web Page (filtered) exports and also performs some content character substitutions. I'll put a pull request together for you over the next week or so.

I came across another MS Word open issue #855... there could be others...
Not sure that's related. Looks like a new XML "smart tags" name-space was introduced which HTMLTidy wasn't prepared for. There have to be XML parsers out there which can strip these out.

I am open to fixing issues where and when they're reported
I know :) I get it. I don't think it's possible to pin this one down without knowing the use case for bare substituting the characters. I haven't checked the code (my C and C++ are rusty), but the substitution code will have been put there for a reason and you can bet someone is relying on the behaviour. I may take a closer look at the code at some point in the future.

Thanks again.

[geoffmcl]

@JustAGuyCoding thank you for the further feedback, comments, replies...

First I would take quite an opposite view -

I suspect you're supposed to use one option or the other to ...

No! I see word-2000 means exactly that! Read the primary code -

int         tidyDocCleanAndRepair( TidyDocImpl* doc )
{
    ...
    if ( word2K && TY_(IsWord2000)(doc) )
    {
        /* prune Word2000's <![if ...]> ... <![endif]> */
        TY_(DropSections)( doc, &doc->root );

        /* drop style & class attributes and empty p, span elements */
        TY_(CleanWord2000)( doc, &doc->root );
        TY_(DropEmptyElements)(doc, &doc->root);
    }

And it is IMPORTANT to notice the TY_(IsWord2000)(doc) service, which has two checks for a word-2000 doc -

1. On html tag, any attribute `xmlns:o`, and if not, then 
2. Does the head meta generator contain `Microsoft`, ignoring case...

But then in CleanWord2000 service, we get another check, not quite the same -

1. On the html tag, is the **first** attrib `xmlns:o`, and if not then 
2. Is `bare` option **on*... 

This is only to determine if the html tag attribute should be discarded...

I think this is not quite right... confusing even... and leads you, and maybe others, to somehow relate bare to word-2000...

Read the code comment, and I would suggest a patch something like -

diff --git a/src/clean.c b/src/clean.c
index e96dd3f..059e9da 100644
--- a/src/clean.c
+++ b/src/clean.c
@@ -1890,8 +1890,7 @@ void TY_(CleanWord2000)( TidyDocImpl* doc, Node *node)
         if ( nodeIsHTML(node) )
         {
             /* check that it's a Word 2000 document */
-            if ( !TY_(GetAttrByName)(node, "xmlns:o") &&
-                 !cfgBool(doc, TidyMakeBare) )
+            if ( !TY_(IsWord2000) (doc) )
                 return;
 
             /* Output proprietary attributes to maintain errout compatability

Or that check removed entirely...

In other words, if word-2000 is on, on the html tag, always show the word-2000 warnings ... and delete the html attributes... regardless...

At worst, a valid say <html lang="en">, or others, will be dropped...

Does that work with your filtered export? Do they still have a head meta generator Microsoft signature?

If not, can you identify what Tidy could reliably use... for Is this reasonably a MSWord doc?... or definitively...

So I repeat, I do NOT think this is in any way related to the bare option...

You seem to suggest you do not know the use case for this option, but it is absolutely all there, in the code...

Docs are sometimes written by people, who have not fully read, or understood, the code, or have forgotten, or etc...

I still suggest the current docs be adjusted as proposed... to make it clear what's what...

Of course, sometimes the opposite could be true... the docs truly reflect the intention of the code... which, if true, means the code needs to be changed to exactly match...

In totality, I just do NOT see that being the case here... I may be wrong... please explain...

Now, all that means is that if you do "tidy -config cfg_885.txt in_885.html" you will get some surprises in the output...

Maybe these are not acceptable, then I look forward to other suggestions, code, patches,... on how to improve Tidy...

Look forward to further feedback, comments, patches, PR, etc... or a new issue... thanks...