Simplify feed ID generation and support URL fragments

[Scheirle]

Changes:

• Simplified the unique_id generation of a feed item
  • The new ids contain now two / more, one at the beginning of the path section and one at the end.
  • The new ids also support now url fragments (e.g. An entry located at http://test.de/index.html#fragment gets now the id tag:test.de,{date}:/index.html/fragment instead of tag:test.de,{date}:index.html)
• Updated unittest output to match the new ids

TODO:

• Update get_tag_uri: Changes get_tag_uri so that the '/' before a fragment gets only added if there is a fragment feedgenerator#4
• Wait for Add updated date to feed using modified. Fix #1933 #1937 (no technical reason; it's just easier for everyone)
• Wait for feedgenerator 1.9
• Update the test output with the new ids

[justinmayer]

Hi @Scheirle. Thanks for the contribution. I hadn't realized that the feedgenerator library has its own get_tag_uri function. That would have made my job quite a bit easier. :^)

That said, my original implementation produces the precise tag I'm looking for:

<id>tag:hackercodex.com,2013-10-30:guide/python-development-environment-on-mac-osx/</id>

... whereas the implementation in this pull request produces:

<id>tag:hackercodex.com,2013-10-30:/guide/python-development-environment-on-mac-osx//</id>

What is the benefit of prepending and appending a slash at the beginning and end of the URL?

If the extra slashes were kept as implemented in this pull request, at the very least we'd have to ensure the double-slash behavior showcased above is somehow avoided.

[Scheirle]

What is the benefit of prepending and appending a slash at the beginning and end of the URL?

Technically there is no benefit by doing so, but following this guideline to create tag uris you only insert the date between the domain name an the path (Except from the protocol which gets removed.). Therefore all characters of the actual url are still there (especially the first slash).

The slash at the end comes from the lazy implementation of get_tag_uri, which always adds a slash before a possible fragment instead of only when a fragment is there.

To prevent double slashes we could strip the last characters if they are slashes:

...
unique_id=get_tag_uri(link.rstrip('/'), item.date),
...

But keep in mind if the url doesn't have a slash at the end. E.g http://test.de/index.html This modification still produces: tag:test.de,{date}:/index.html/.

In my opinion it is not worth it to strip the slashes manually, since the only purpose of the id is to be unique which is the case with or without double slashes. And since one reason for this pull request is to make the id generation as simple as possible I think we should rely on the get_tag_uri function.

But if you want me to update the pull request with the above modification I am pleased to do so.

[Scheirle]

@justinmayer any thoughts on this?

---

The pelican comment system depends on the support of url fragments, so it can generates valid feed entry ids.

[justinmayer]

@Scheirle: What do you think about modifying get_tag_uri such that it only adds a slash when it's appropriate to do so?

Please accept my sincere apologies for the absurdly long delay in responding!

[Scheirle]

What do you think about modifying get_tag_uri such that it only adds a slash when it's appropriate to do so?

Yeah that is probably the right way to do it.

[justinmayer]

Corresponding feedgenerator pull request merged!

[justinmayer]

Checked off another box in the above list. 🎊

[Scheirle]

Finally this can be merged!

• feedgenerator 1.9 got released
• I updated the test output

It would save work (Avoid further merge conflicts) if this gets merged before #1989 resolves it merge conflicts.

[justinmayer]

Excellent work, Bernhard!

[justinmayer]

I have tested this, and it behaves as expected. Any other @getpelican/reviewers care to review this before it is merged?

[ionelmc]

Note that this causes a regression - different ids will be generated if you upgrade pelican!

[ionelmc]

This means all your items will get republished wherever they are consumed, an undesirable consequence (annoying feed spam).