Check .shtml files for noindex directives

[TravisBrace]

Summary

Checks .shtml files for noindex directives in the head.

Hoping adding "shtml" would help treat index.shtml pages the same as index.html pages removing the from sitemap so pages end in /

Closing Issues

Closes #51

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Improvements to existing code, such as refactoring or optimizations (non-breaking)
• Breaking change (fix or feature that would cause existing functionality to change)
• I have read the CONTRIBUTING document.
• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have added tests to cover my changes.
• All new and existing tests passed.

[cicirello]

That's part of it. That alone will just cause the action to search for noindex directives within shtml files, excluding those that have it. Without your change, it currently just looks for noindex in files with extension html or htm. The index.html is dropped from the url somewhere else.

[TravisBrace]

Oh nice! that would definitely help out for some of the older .shtml sites we keep on github.

Is this the other part here? I could add a

elif f == "index.shtml" :
return ""

To the pull request also.

def sortname(f, dropExtension=False) :
    """Partial url to sort by, which strips out the filename
    if the filename is index.html.
    Keyword arguments:
    f - Filename with path
    dropExtension - true to drop extensions of .html from the filename when sorting
    """
    if len(f) >= 11 and f[-11:] == "/index.html" :
        return f[:-10]
    elif f == "index.html" :
        return ""
    elif dropExtension and len(f) >= 5 and f[-5:] == ".html" :
        return f[:-5]
    else :
        return f

[cicirello]

@TravisBrace Yes, the sortname function is where that is handled. Or at least now it is. While looking at code to confirm your question I noticed I had some redundancy in there. I just committed a change to your PR to remove the redundancy so that the other place with that logic now just calls the sortname function.

I think you should now be able to add the elif you mentioned. You might actually need two of them. You'll notice the first if checks if the file path ends with "/index.html" and the elif after that checks if the path equals "index.html" (no slash). If I remember correctly, the reason it's not simply checking if it ends with "index.html" is for the possibly unlikely case of a filename like "somethingindex.html".

[cicirello]

@TravisBrace I'm going to merge this now since it handles checking head of .shtml files for noindex directives. I created an issue for the thing you really want, dropping index.shtml from URLs. Feel free to submit an additional PR for that one. It should just involve changing the sortname function.