Py3k v4 #618

Closed
wants to merge 18 commits into
from

Conversation

Projects
None yet
7 participants
@dmdm
Contributor

dmdm commented Dec 4, 2012

This is a port of the latest Pelican 3.1.1.

Note: Some external libraries must also be updated for Python 3.x compatibility; see README-PY3K for details.

Issues:

  • Travis cannot yet test for Python 3.2 because py3k versions of external libs are needed
  • Py26 is broken. Creation of "locale_date" uses method "encode()" with arguments that did not exist in Python 2.6.
@@ -11,3 +11,5 @@ tags
.tox
.coverage
htmlcov
+six-*.egg/

This comment has been minimized.

Show comment Hide comment
@almet

almet Dec 4, 2012

Owner

we can actually ignore all the .egg I think, not only six.

@almet

almet Dec 4, 2012

Owner

we can actually ignore all the .egg I think, not only six.

@@ -11,3 +11,5 @@ tags
.tox
.coverage
htmlcov
+six-*.egg/
+*.orig

This comment has been minimized.

Show comment Hide comment
@almet

almet Dec 4, 2012

Owner

what is this?

@almet

almet Dec 4, 2012

Owner

what is this?

This comment has been minimized.

Show comment Hide comment
@justinmayer

justinmayer Dec 4, 2012

Owner

Yeah, I'm guessing this should be in local devs' .gitignore and not in the repo's.

@justinmayer

justinmayer Dec 4, 2012

Owner

Yeah, I'm guessing this should be in local devs' .gitignore and not in the repo's.

INSTALL.rst
+The requirements in setup.py list 'feedgenerator'. For Python 2 pip will fetch
+it from the cheeseshop for you. However, for Python 3 you need the Py3k port of
+feedgenerator. Currently it is available only here:
+https://github.com/dmdm/feedgenerator-py3k/tree/from_django_1.5.dev20120824122350

This comment has been minimized.

Show comment Hide comment
@almet

almet Dec 4, 2012

Owner

is it compatible for python 3 and py 2.X series? I'm the owner of the "feedgenerator" module, I can probably do something for it? (like releasing a py2 and py3 compatible version)

@almet

almet Dec 4, 2012

Owner

is it compatible for python 3 and py 2.X series? I'm the owner of the "feedgenerator" module, I can probably do something for it? (like releasing a py2 and py3 compatible version)

This comment has been minimized.

Show comment Hide comment
@dmdm

dmdm Dec 4, 2012

Contributor

It is. Some time ago we talked about that on IRC, but I have not yet found the time to prepare a PyPi package.

@dmdm

dmdm Dec 4, 2012

Contributor

It is. Some time ago we talked about that on IRC, but I have not yet found the time to prepare a PyPi package.

This comment has been minimized.

Show comment Hide comment
@almet

almet Dec 4, 2012

Owner

okay, I thnk this should come with the next pelican release then.

@almet

almet Dec 4, 2012

Owner

okay, I thnk this should come with the next pelican release then.

README-PY3K
@@ -0,0 +1,80 @@
+This is a port of Pelican to Python 3.

This comment has been minimized.

Show comment Hide comment
@almet

almet Dec 4, 2012

Owner

maybe this could go to a blogpost on blog.getpelican.com?

@almet

almet Dec 4, 2012

Owner

maybe this could go to a blogpost on blog.getpelican.com?

This comment has been minimized.

Show comment Hide comment
@dmdm

dmdm Dec 4, 2012

Contributor

Avec plaisir. How? Maybe the text should be rewritten in cleaner written language then...

@dmdm

dmdm Dec 4, 2012

Contributor

Avec plaisir. How? Maybe the text should be rewritten in cleaner written language then...

This comment has been minimized.

Show comment Hide comment
@almet

almet Dec 4, 2012

Owner

issue a pull request with this at getpelican/pelican-blog

@almet

almet Dec 4, 2012

Owner

issue a pull request with this at getpelican/pelican-blog

README-PY3K
+had to be changed (grammar ok? ;)) for Py3k. Unfortunately, I was not able to decipher
+the purpose of those lines, so I just commented them out. Hah, if we believe the test
+suite, no bugs were introduced by this.
+==> Behaviour on Win32 needs testing, since the commented out block contains a line

This comment has been minimized.

Show comment Hide comment
@almet

almet Dec 4, 2012

Owner

is this still valid?

@almet

almet Dec 4, 2012

Owner

is this still valid?

This comment has been minimized.

Show comment Hide comment
@dmdm

dmdm Dec 4, 2012

Contributor

Not really. Couple of days ago, tehkonst reported an encoding glitch in Russian, which triggered me to investigate. I think, I solved it, but wanted to wait for tehkonst's response before incorporating that in Pelican.

@dmdm

dmdm Dec 4, 2012

Contributor

Not really. Couple of days ago, tehkonst reported an encoding glitch in Russian, which triggered me to investigate. I think, I solved it, but wanted to wait for tehkonst's response before incorporating that in Pelican.

This comment has been minimized.

Show comment Hide comment
@almet

almet Dec 4, 2012

Owner

okay, good. I'll let you finish your work on this then

@almet

almet Dec 4, 2012

Owner

okay, good. I'll let you finish your work on this then

@@ -1,8 +1,9 @@
# Tests
-unittest2
+unittest2py3k

This comment has been minimized.

Show comment Hide comment
@almet

almet Dec 4, 2012

Owner

is this valid for 2.x and 3.x ?

@almet

almet Dec 4, 2012

Owner

is this valid for 2.x and 3.x ?

This comment has been minimized.

Show comment Hide comment
@dmdm

dmdm Dec 4, 2012

Contributor

Assumed so. That's the name of the PyPi package, it installs as "unittest2", like the unittest2 package does. But since now you ask, maybe we should investigate further.

@dmdm

dmdm Dec 4, 2012

Contributor

Assumed so. That's the name of the PyPi package, it installs as "unittest2", like the unittest2 package does. But since now you ask, maybe we should investigate further.

mock
# Optional Packages
Markdown
-BeautifulSoup
+BeautifulSoup4
+lxml

This comment has been minimized.

Show comment Hide comment
@almet

almet Dec 4, 2012

Owner

why do we need to depend on lxml explicitely?

@almet

almet Dec 4, 2012

Owner

why do we need to depend on lxml explicitely?

This comment has been minimized.

Show comment Hide comment
@dmdm

dmdm Dec 4, 2012

Contributor

Not sure if bs4 pulls it implicitly.

@dmdm

dmdm Dec 4, 2012

Contributor

Not sure if bs4 pulls it implicitly.

@@ -119,7 +120,7 @@ def _update_context(self, items):
for item in items:
value = getattr(self, item)
if hasattr(value, 'items'):
- value = value.items()
+ value = list(value.items())

This comment has been minimized.

Show comment Hide comment
@almet

almet Dec 4, 2012

Owner

why is that?

@almet

almet Dec 4, 2012

Owner

why is that?

This comment has been minimized.

Show comment Hide comment
@dmdm

dmdm Dec 4, 2012

Contributor

Because items() in Py2 returns a list, in Py3 an iterator. In many circumstances this difference is negligible, but sometimes it crashes. Explicitly converting the return value of items() to a list is a safeguard (e.g. 2to3 does this always).

@dmdm

dmdm Dec 4, 2012

Contributor

Because items() in Py2 returns a list, in Py3 an iterator. In many circumstances this difference is negligible, but sometimes it crashes. Explicitly converting the return value of items() to a list is a safeguard (e.g. 2to3 does this always).

This comment has been minimized.

Show comment Hide comment
@hukka

hukka Dec 14, 2012

IMO it would be better to make tests that catch these, and change the whole Pelican into iterator-resistant. Or are there any particular reasons it needs real lists?

Of course I do realize that would be more work, but there should be at least a comment here mentioning that this is just a safeguard, a real list might not be needed. Otherwise someone in few years won't dare to change that list into an iterator, since it looks like it's intentionally like that.

@hukka

hukka Dec 14, 2012

IMO it would be better to make tests that catch these, and change the whole Pelican into iterator-resistant. Or are there any particular reasons it needs real lists?

Of course I do realize that would be more work, but there should be at least a comment here mentioning that this is just a safeguard, a real list might not be needed. Otherwise someone in few years won't dare to change that list into an iterator, since it looks like it's intentionally like that.

This comment has been minimized.

Show comment Hide comment
@almet

almet Dec 14, 2012

Owner

That's right, but let's occult this for now, I think.

@almet

almet Dec 14, 2012

Owner

That's right, but let's occult this for now, I think.

This comment has been minimized.

Show comment Hide comment
@dmdm

dmdm Dec 19, 2012

Contributor

I must correct myself, keys() in Python 3 does not return an iterator, but a so-called dictionary view (which is a new entity in py3k). However, what is said about those being a possible source of errors later in the code is still true. Besides not being copyable, those views also are not indexable [#fn1]_; i.o.w. like iterators, they do not behave like lists.

Like tpievila, I too like to vote for removing list() here. My thinking is this: we here, at the source of some data, cannot possibly imagine in which circumstances this data will be used by other parts of a program. Therefore we should keep the data here in its native type, and leave it to the other code to handle it properly. If some code e.g. needs to index that data and fails because of the TypeError, that code is to be refactored to cast the data into a suitable format, and this change should be kept local to that part.

I'm aware that Alexis announced an eclipse of this for now; am just thinking aloud ;) Comments welcome.

.. [#fn1] Fed the return value of keys() into PrettyTable() as column headers, and got TypeError: 'KeysView' object does not support indexing

@dmdm

dmdm Dec 19, 2012

Contributor

I must correct myself, keys() in Python 3 does not return an iterator, but a so-called dictionary view (which is a new entity in py3k). However, what is said about those being a possible source of errors later in the code is still true. Besides not being copyable, those views also are not indexable [#fn1]_; i.o.w. like iterators, they do not behave like lists.

Like tpievila, I too like to vote for removing list() here. My thinking is this: we here, at the source of some data, cannot possibly imagine in which circumstances this data will be used by other parts of a program. Therefore we should keep the data here in its native type, and leave it to the other code to handle it properly. If some code e.g. needs to index that data and fails because of the TypeError, that code is to be refactored to cast the data into a suitable format, and this change should be kept local to that part.

I'm aware that Alexis announced an eclipse of this for now; am just thinking aloud ;) Comments welcome.

.. [#fn1] Fed the return value of keys() into PrettyTable() as column headers, and got TypeError: 'KeysView' object does not support indexing

@tehkonst

This comment has been minimized.

Show comment Hide comment
@tehkonst

tehkonst Dec 6, 2012

Contributor

@dmdm,
Please, see my commit - fix encoding

Contributor

tehkonst commented Dec 6, 2012

@dmdm,
Please, see my commit - fix encoding

Dirk Makowski added some commits Dec 7, 2012

Dirk Makowski
Deactivated py32 for travis
Testing Pelican on Python 3 requires that all tested external libraries
are installed in their py3k compatible incarnation (see README-PY3K for
details).

Maybe we can instruct travis to pull them from github and install them.
I do not know that yet.
+ # "Øl trinken beim Besäufnis"
+ # --> "Øl trinken beim Besäufnis"
+ date_format = date_format.encode('ascii',
+ errors="xmlcharrefreplace")

This comment has been minimized.

Show comment Hide comment
@dmdm

dmdm Dec 8, 2012

Contributor

Keyword argument "errors" breaks compatibility with Python 2.6

@dmdm

dmdm Dec 8, 2012

Contributor

Keyword argument "errors" breaks compatibility with Python 2.6

+ # But with "xmlcharrefreplace" our formatted date will produce
+ # *yuck* like this:
+ # "Øl trinken beim Besäufnis"
+ # --> "Øl trinken beim Besäufnis"

This comment has been minimized.

Show comment Hide comment
@tehkonst

tehkonst Dec 8, 2012

Contributor

What to do if we want to write in the file string "Øl trinken beim Besäufnis" and not Øl trinken beim Besäufnis?
If we write

# -*- coding: utf-8 -*-
from codecs import open
f = open('test.log', 'w', 'utf-8')
f.write( "Øl trinken beim Besäufnis".decode('utf-8') )

it works. But if we add from __future__ import unicode_literals, it doesn't. So I think we don't need encode ascii.

And is strange that when code is running in Pydev, it has not UnicodeEncodeError with not ASCII.

@tehkonst

tehkonst Dec 8, 2012

Contributor

What to do if we want to write in the file string "Øl trinken beim Besäufnis" and not Øl trinken beim Besäufnis?
If we write

# -*- coding: utf-8 -*-
from codecs import open
f = open('test.log', 'w', 'utf-8')
f.write( "Øl trinken beim Besäufnis".decode('utf-8') )

it works. But if we add from __future__ import unicode_literals, it doesn't. So I think we don't need encode ascii.

And is strange that when code is running in Pydev, it has not UnicodeEncodeError with not ASCII.

This comment has been minimized.

Show comment Hide comment
@dmdm

dmdm Dec 8, 2012

Contributor

encode ascii here is needed for strftime(), not for writing into a file.

On Python 2, this works: fmt is a utf-8 encoded byte string

# -*- coding: utf-8 -*-
from __future__ import print_function
import datetime
fmt = "Øl trinken beim Besäufnis in the year %Y"
d = datetime.date(2012, 12, 8)
print(d.strftime(fmt))

However, this does not: fmt is unicode now

# -*- coding: utf-8 -*-
from __future__ import print_function, unicode_literals
import datetime
fmt = "Øl trinken beim Besäufnis in the year %Y"
d = datetime.date(2012, 12, 8)
print(d.strftime(fmt))

Therefore, fmt has to be encoded.


Writing into a file, I get these results:
Encoding error without unicode_literals

# -*- coding: utf-8 -*-
from __future__ import print_function
import codecs
s = "Øl trinken beim Besäufnis"
with codecs.open("test.txt", "w", encoding="utf-8") as fh:
    fh.write(s)

Works fine with unicode_literals:

# -*- coding: utf-8 -*-
from __future__ import print_function, unicode_literals
import codecs
s = "Øl trinken beim Besäufnis"
with codecs.open("test.txt", "w", encoding="utf-8") as fh:
    fh.write(s)

Without unicode_literals, a utf-8 source code literal (which is a byte string) has to be _de_coded into unicode, before it can be written into a stream that expects to _en_code its input:

# -*- coding: utf-8 -*-
from __future__ import print_function
import codecs
s = "Øl trinken beim Besäufnis"
with codecs.open("test.txt", "w", encoding="utf-8") as fh:
    fh.write(s.decode('utf-8'))

This is what you had posted.

@dmdm

dmdm Dec 8, 2012

Contributor

encode ascii here is needed for strftime(), not for writing into a file.

On Python 2, this works: fmt is a utf-8 encoded byte string

# -*- coding: utf-8 -*-
from __future__ import print_function
import datetime
fmt = "Øl trinken beim Besäufnis in the year %Y"
d = datetime.date(2012, 12, 8)
print(d.strftime(fmt))

However, this does not: fmt is unicode now

# -*- coding: utf-8 -*-
from __future__ import print_function, unicode_literals
import datetime
fmt = "Øl trinken beim Besäufnis in the year %Y"
d = datetime.date(2012, 12, 8)
print(d.strftime(fmt))

Therefore, fmt has to be encoded.


Writing into a file, I get these results:
Encoding error without unicode_literals

# -*- coding: utf-8 -*-
from __future__ import print_function
import codecs
s = "Øl trinken beim Besäufnis"
with codecs.open("test.txt", "w", encoding="utf-8") as fh:
    fh.write(s)

Works fine with unicode_literals:

# -*- coding: utf-8 -*-
from __future__ import print_function, unicode_literals
import codecs
s = "Øl trinken beim Besäufnis"
with codecs.open("test.txt", "w", encoding="utf-8") as fh:
    fh.write(s)

Without unicode_literals, a utf-8 source code literal (which is a byte string) has to be _de_coded into unicode, before it can be written into a stream that expects to _en_code its input:

# -*- coding: utf-8 -*-
from __future__ import print_function
import codecs
s = "Øl trinken beim Besäufnis"
with codecs.open("test.txt", "w", encoding="utf-8") as fh:
    fh.write(s.decode('utf-8'))

This is what you had posted.

This comment has been minimized.

Show comment Hide comment
@tehkonst

tehkonst Dec 8, 2012

Contributor

Ok, I understand. But why do you use encode('ascii', errors="xmlcharrefreplace")?
More logically to use encode('utf-8').
Then symbols like "Ø" will be not replaced.

@tehkonst

tehkonst Dec 8, 2012

Contributor

Ok, I understand. But why do you use encode('ascii', errors="xmlcharrefreplace")?
More logically to use encode('utf-8').
Then symbols like "Ø" will be not replaced.

This comment has been minimized.

Show comment Hide comment
@tehkonst

tehkonst Dec 8, 2012

Contributor

I see, there is one more locale encoding. Too difficult.

@tehkonst

tehkonst Dec 8, 2012

Contributor

I see, there is one more locale encoding. Too difficult.

This comment has been minimized.

Show comment Hide comment
@dmdm

dmdm Dec 8, 2012

Contributor

Yes. As long as your current locale is UTF-8 encoded. If your locale is e.g. 'C', encoding with UTF-8 is not sufficient. In the strftime test case replace encode('ascii', errors="xmlcharrefreplace") with encode('utf-8'). You will notice a failure like this:

File "/home/dm/myprojects/pelican3.0-pyk3-devel-env/src/my/test_strftime/tests/test_strftime.py", line 119, in test_strftime
    fmt("C", "%c Øl trinken beim Besäufnis", "Fri Mar  2 00:00:00 2012 Øl trinken beim Besäufnis")
  File "/home/dm/myprojects/pelican3.0-pyk3-devel-env/src/my/test_strftime/tests/test_strftime.py", line 53, in fmt
    result = unicode(result)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 25: ordinal not in range(128)
@dmdm

dmdm Dec 8, 2012

Contributor

Yes. As long as your current locale is UTF-8 encoded. If your locale is e.g. 'C', encoding with UTF-8 is not sufficient. In the strftime test case replace encode('ascii', errors="xmlcharrefreplace") with encode('utf-8'). You will notice a failure like this:

File "/home/dm/myprojects/pelican3.0-pyk3-devel-env/src/my/test_strftime/tests/test_strftime.py", line 119, in test_strftime
    fmt("C", "%c Øl trinken beim Besäufnis", "Fri Mar  2 00:00:00 2012 Øl trinken beim Besäufnis")
  File "/home/dm/myprojects/pelican3.0-pyk3-devel-env/src/my/test_strftime/tests/test_strftime.py", line 53, in fmt
    result = unicode(result)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 25: ordinal not in range(128)
+ else:
+ result = unicode(result)
+ # Convert XML character references back to unicode characters.
+ if "&#" in result:

This comment has been minimized.

Show comment Hide comment
@dmdm

dmdm Dec 8, 2012

Contributor

Btw, this converts ""&#216l trinken beim Bes&#228ufnis" back to unicoded "Øl trinken beim Besäufnis"

@dmdm

dmdm Dec 8, 2012

Contributor

Btw, this converts ""&#216l trinken beim Bes&#228ufnis" back to unicoded "Øl trinken beim Besäufnis"

@tehkonst

This comment has been minimized.

Show comment Hide comment
@tehkonst

tehkonst Dec 10, 2012

Contributor

On python 2.7 Detected misconfiguration with OUTPUT_SOURCES_EXTENSION. It's from here

Contributor

tehkonst commented Dec 10, 2012

On python 2.7 Detected misconfiguration with OUTPUT_SOURCES_EXTENSION. It's from here

@dmdm

This comment has been minimized.

Show comment Hide comment
@dmdm

dmdm Dec 10, 2012

Contributor

Thanks for spotting this

Contributor

dmdm commented Dec 10, 2012

Thanks for spotting this

@justinmayer

This comment has been minimized.

Show comment Hide comment
@justinmayer

justinmayer Dec 16, 2012

Owner

Doing some testing. My steps:

brew install python3 --with-brewed-openssl
virtualenv -p python3 pelican-python3
cd pelican-python3
. bin/activate
mkdir src
git clone https://github.com/getpelican/pelican.git $VIRTUAL_ENV/src/pelican
cd src/pelican
git remote add dmdm https://github.com/dmdm/pelican.git
git fetch dmdm py3k-v4:py3k-v4
git checkout py3k-v4
python setup.py develop
pip install -r dev_requirements.txt

The above is normally enough. Not here. Had to manually install all of the following:

pip install docutils pygments jinja2
pip install https://github.com/dmdm/feedgenerator-py3k/archive/master.zip

Generate the sample output:

pelican samples/content -s samples/pelican.conf.py -o ../output

Error:

CRITICAL: decoding str is not supported
Traceback (most recent call last):
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/bin/pelican", line 9, in <module>
    load_entry_point('pelican==3.2', 'console_scripts', 'pelican')()
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/src/pelican/pelican/__init__.py", line 324, in main
    pelican.run()
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/src/pelican/pelican/__init__.py", line 169, in run
    p.generate_output(writer)
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/src/pelican/pelican/generators.py", line 431, in generate_output
    self.generate_feeds(writer)
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/src/pelican/pelican/generators.py", line 192, in generate_feeds
    self.settings['FEED_ALL_ATOM'])
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/src/pelican/pelican/writers.py", line 73, in write_feed
    self._add_item_to_the_feed(feed, elements[i])
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/src/pelican/pelican/writers.py", line 47, in _add_item_to_the_feed
    self.settings.get('TIMEZONE', None)))
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/lib/python3.3/site-packages/feedgenerator.py", line 224, in add_item
    'author_name': to_unicode(author_name),
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/lib/python3.3/site-packages/feedgenerator.py", line 213, in <lambda>
    to_unicode = lambda s: force_unicode(s, strings_only=True)
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/lib/python3.3/site-packages/feedgenerator.py", line 104, in force_unicode
    s = str(str(s), encoding, errors)
TypeError: decoding str is not supported
Owner

justinmayer commented Dec 16, 2012

Doing some testing. My steps:

brew install python3 --with-brewed-openssl
virtualenv -p python3 pelican-python3
cd pelican-python3
. bin/activate
mkdir src
git clone https://github.com/getpelican/pelican.git $VIRTUAL_ENV/src/pelican
cd src/pelican
git remote add dmdm https://github.com/dmdm/pelican.git
git fetch dmdm py3k-v4:py3k-v4
git checkout py3k-v4
python setup.py develop
pip install -r dev_requirements.txt

The above is normally enough. Not here. Had to manually install all of the following:

pip install docutils pygments jinja2
pip install https://github.com/dmdm/feedgenerator-py3k/archive/master.zip

Generate the sample output:

pelican samples/content -s samples/pelican.conf.py -o ../output

Error:

CRITICAL: decoding str is not supported
Traceback (most recent call last):
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/bin/pelican", line 9, in <module>
    load_entry_point('pelican==3.2', 'console_scripts', 'pelican')()
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/src/pelican/pelican/__init__.py", line 324, in main
    pelican.run()
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/src/pelican/pelican/__init__.py", line 169, in run
    p.generate_output(writer)
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/src/pelican/pelican/generators.py", line 431, in generate_output
    self.generate_feeds(writer)
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/src/pelican/pelican/generators.py", line 192, in generate_feeds
    self.settings['FEED_ALL_ATOM'])
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/src/pelican/pelican/writers.py", line 73, in write_feed
    self._add_item_to_the_feed(feed, elements[i])
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/src/pelican/pelican/writers.py", line 47, in _add_item_to_the_feed
    self.settings.get('TIMEZONE', None)))
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/lib/python3.3/site-packages/feedgenerator.py", line 224, in add_item
    'author_name': to_unicode(author_name),
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/lib/python3.3/site-packages/feedgenerator.py", line 213, in <lambda>
    to_unicode = lambda s: force_unicode(s, strings_only=True)
  File "/Volumes/Koneko/Virtualenvs/pelican-python3/lib/python3.3/site-packages/feedgenerator.py", line 104, in force_unicode
    s = str(str(s), encoding, errors)
TypeError: decoding str is not supported
@dmdm

This comment has been minimized.

Show comment Hide comment
@dmdm

dmdm Dec 17, 2012

Contributor

The error is created by the Py2 version of feedgenerator. Have updated the repo feedgenerator-py3k so that master now has the py3k code. (Previously it was only in the from_django...-branch.)

Installation in a fresh virtualenv works for me like this: https://gist.github.com/4316576

That the installation of feedgenerator fails is expected, the PyPi package contains only Py2 code. For Py3k there's not yet a PyPi package available. So feedgenerator-py3k has to be manually installed from the github repo.

Unexpectedly, I had to manually pip-install markdown. It is missing from the requires list in setup.py. This is also the case in the original Pelican master. Shall I add it?

I wonder why you had to manually install docutils, pygments and jinja...

Btw, the feed links in the generated sample blog point to the notmyidea website online (and produce a 404), not to the locally generated feeds as I had expected. Is that intended behaviour?

Contributor

dmdm commented Dec 17, 2012

The error is created by the Py2 version of feedgenerator. Have updated the repo feedgenerator-py3k so that master now has the py3k code. (Previously it was only in the from_django...-branch.)

Installation in a fresh virtualenv works for me like this: https://gist.github.com/4316576

That the installation of feedgenerator fails is expected, the PyPi package contains only Py2 code. For Py3k there's not yet a PyPi package available. So feedgenerator-py3k has to be manually installed from the github repo.

Unexpectedly, I had to manually pip-install markdown. It is missing from the requires list in setup.py. This is also the case in the original Pelican master. Shall I add it?

I wonder why you had to manually install docutils, pygments and jinja...

Btw, the feed links in the generated sample blog point to the notmyidea website online (and produce a 404), not to the locally generated feeds as I had expected. Is that intended behaviour?

@justinmayer

This comment has been minimized.

Show comment Hide comment
@justinmayer

justinmayer Dec 17, 2012

Owner

Thanks for the help, Dirk. Uninstalling feedgenerator and re-installing from the newly-updated master branch did indeed rectify the aforementioned error.

Unexpectedly, I had to manually pip-install markdown. It is missing from the requires list in setup.py. This is also the case in the original Pelican master. Shall I add it?

This is the intended behavior. For folks who choose to write their posts in reST format, Markdown is not needed and thus is not currently a hard dependency. That said, Markdown appears to be preferred by a majority of Pelican users, and there are times I wonder whether the current approach makes practical sense.

I wonder why you had to manually install docutils, pygments and jinja

I was confused as well. This is something we should probably investigate further.

Btw, the feed links in the generated sample blog point to the notmyidea website online (and produce a 404), not to the locally generated feeds as I had expected. Is that intended behaviour?

Yes. Links in feeds should always be constructed with absolute URLs; it's too easy to produce incorrect feed links if they are relatively generated. The W3C Feed Validator will produce warnings when relative href links are present. More info here:

http://www.intertwingly.net/blog/1314.html
http://www.rimmkaufman.com/blog/absolute-urls/19012007/

Owner

justinmayer commented Dec 17, 2012

Thanks for the help, Dirk. Uninstalling feedgenerator and re-installing from the newly-updated master branch did indeed rectify the aforementioned error.

Unexpectedly, I had to manually pip-install markdown. It is missing from the requires list in setup.py. This is also the case in the original Pelican master. Shall I add it?

This is the intended behavior. For folks who choose to write their posts in reST format, Markdown is not needed and thus is not currently a hard dependency. That said, Markdown appears to be preferred by a majority of Pelican users, and there are times I wonder whether the current approach makes practical sense.

I wonder why you had to manually install docutils, pygments and jinja

I was confused as well. This is something we should probably investigate further.

Btw, the feed links in the generated sample blog point to the notmyidea website online (and produce a 404), not to the locally generated feeds as I had expected. Is that intended behaviour?

Yes. Links in feeds should always be constructed with absolute URLs; it's too easy to produce incorrect feed links if they are relatively generated. The W3C Feed Validator will produce warnings when relative href links are present. More info here:

http://www.intertwingly.net/blog/1314.html
http://www.rimmkaufman.com/blog/absolute-urls/19012007/

@justinmayer

This comment has been minimized.

Show comment Hide comment
@justinmayer

justinmayer Dec 17, 2012

Owner

In my testing, I came across a Python 3 compatibility issue with SmartyPants. Changing line 860 from...

print docstring_html

... to ...

print(docstring_html)

... appears to rectify the problem. Looking at the PyPI entry for SmartyPants, it seems @hlian is the maintainer for the PyPI-packaged version. I sent a message to Hao via email and requested a new SmartyPants PyPI release containing this change.

Owner

justinmayer commented Dec 17, 2012

In my testing, I came across a Python 3 compatibility issue with SmartyPants. Changing line 860 from...

print docstring_html

... to ...

print(docstring_html)

... appears to rectify the problem. Looking at the PyPI entry for SmartyPants, it seems @hlian is the maintainer for the PyPI-packaged version. I sent a message to Hao via email and requested a new SmartyPants PyPI release containing this change.

@dmdm

This comment has been minimized.

Show comment Hide comment
@dmdm

dmdm Dec 17, 2012

Contributor

Why did you not use the py3k versions of the external libs, as described in the readme-py3k?

Contributor

dmdm commented Dec 17, 2012

Why did you not use the py3k versions of the external libs, as described in the readme-py3k?

@justinmayer

This comment has been minimized.

Show comment Hide comment
@justinmayer

justinmayer Dec 17, 2012

Owner

Thanks for the info. I did not realize that (a) the readme-py3k doc contained information necessary to use external libraries and (b) you had already prepared Python 3.x-compatible versions of those external libraries. I updated the PR description to make that a bit clearer.

Owner

justinmayer commented Dec 17, 2012

Thanks for the info. I did not realize that (a) the readme-py3k doc contained information necessary to use external libraries and (b) you had already prepared Python 3.x-compatible versions of those external libraries. I updated the PR description to make that a bit clearer.

@justinmayer

This comment has been minimized.

Show comment Hide comment
@justinmayer

justinmayer Dec 19, 2012

Owner

I sent a message to Chad Miller, author of SmartyPants.py, and asked him if he would issue a new release containing the one-line change necessary for Python 3 compatibility. He replied immediately and said he would. A true mensch and paragon for project maintainers everywhere! 😄

Owner

justinmayer commented Dec 19, 2012

I sent a message to Chad Miller, author of SmartyPants.py, and asked him if he would issue a new release containing the one-line change necessary for Python 3 compatibility. He replied immediately and said he would. A true mensch and paragon for project maintainers everywhere! 😄

@justinmayer

This comment has been minimized.

Show comment Hide comment
@justinmayer

justinmayer Dec 19, 2012

Owner

Wow. Chad works fast. He's already issued a new release containing the Python 3 fix, as well as a tag-in-SGML-comment bug fix:

http://web.chad.org/projects/smartypants.py/smartypants.py_1.5_1.7

I've also taken the liberty of asking @hlian if he would be so kind as to issue a new release on PyPI.

Owner

justinmayer commented Dec 19, 2012

Wow. Chad works fast. He's already issued a new release containing the Python 3 fix, as well as a tag-in-SGML-comment bug fix:

http://web.chad.org/projects/smartypants.py/smartypants.py_1.5_1.7

I've also taken the liberty of asking @hlian if he would be so kind as to issue a new release on PyPI.

@justinmayer

This comment has been minimized.

Show comment Hide comment
@justinmayer

justinmayer Dec 30, 2012

Owner

My understanding is that we're holding off on evaluating other pull requests until we can get Py3K support merged. As the number of outstanding pull requests continues to grow, I wonder whether we can make a coordinated effort to get this blocker approved and merged. That way we can begin going through the pull request backlog.

As I noted previously, I was able to generate content using the Py3K-v4 branch, which behaved as I expected. Have other folks had similar experiences? It seems to me that there are two things to do before this pull request is merged:

  1. reach consensus that this is ready for merging
  2. update the pull request so it can be merged cleanly

Am I missing anything?

Owner

justinmayer commented Dec 30, 2012

My understanding is that we're holding off on evaluating other pull requests until we can get Py3K support merged. As the number of outstanding pull requests continues to grow, I wonder whether we can make a coordinated effort to get this blocker approved and merged. That way we can begin going through the pull request backlog.

As I noted previously, I was able to generate content using the Py3K-v4 branch, which behaved as I expected. Have other folks had similar experiences? It seems to me that there are two things to do before this pull request is merged:

  1. reach consensus that this is ready for merging
  2. update the pull request so it can be merged cleanly

Am I missing anything?

@onlyhavecans

This comment has been minimized.

Show comment Hide comment
@onlyhavecans

onlyhavecans Jan 8, 2013

Owner

I'm doing some from scratch testing and I threw up a temp env and checked out dmdm:py3k-v4 into a fresh folder
However pip install -r dev_requirements.txt failed on unittest2py3k

  File "unittest2/case.py", line 539
    def assertAlmostEqual(self, first, second, *, places=None, msg=None,
                                                ^
SyntaxError: invalid syntax

Should there be a different dev-requirements for py2 now?

Owner

onlyhavecans commented Jan 8, 2013

I'm doing some from scratch testing and I threw up a temp env and checked out dmdm:py3k-v4 into a fresh folder
However pip install -r dev_requirements.txt failed on unittest2py3k

  File "unittest2/case.py", line 539
    def assertAlmostEqual(self, first, second, *, places=None, msg=None,
                                                ^
SyntaxError: invalid syntax

Should there be a different dev-requirements for py2 now?

@almet

This comment has been minimized.

Show comment Hide comment
@almet

almet Jan 11, 2013

Owner

I pushed a working version, from the work by @dmdm at https://github.com/getpelican/pelican/commits/py3k

which is basically this PR with only one commit for the code and one for the docs.

The last thing to do is to add back these tests: http://paste.lolnet.org/mirawugolu.py, ported to unified py2 & 3, and then we're okay to merge!

@justinmayer @bbinet @tbunnyman if one of you can have a look at these commits and check that they look okay, that would be awesome.

Owner

almet commented Jan 11, 2013

I pushed a working version, from the work by @dmdm at https://github.com/getpelican/pelican/commits/py3k

which is basically this PR with only one commit for the code and one for the docs.

The last thing to do is to add back these tests: http://paste.lolnet.org/mirawugolu.py, ported to unified py2 & 3, and then we're okay to merge!

@justinmayer @bbinet @tbunnyman if one of you can have a look at these commits and check that they look okay, that would be awesome.

@bbinet

This comment has been minimized.

Show comment Hide comment
@bbinet

bbinet Jan 11, 2013

Contributor

I won't review this one because I'm not confortable with py3k yet, I need
to learn about py2k/py3k differences first.
But I'm +1 to merge this one soon so that other pull requests can be
reviewed.

On 11 January 2013 03:08, Alexis Metaireau notifications@github.com wrote:

I pushed a working version, from the work by @dmdmhttps://github.com/dmdmat
https://github.com/getpelican/pelican/commits/py3k

which is basically this PR with only one commit for the code: b953900b953900 one for the docs
4d4efc64d4efc6

The last thing to do is to add back these tests:
http://paste.lolnet.org/mirawugolu.py, ported to unified py2 & 3, and
then we're okay to merge!

@justinmayer https://github.com/justinmayer @bbinethttps://github.com/bbinet
@tbunnyman https://github.com/tbunnyman if one of you can have a look
at these commits and check that they look okay, that would be awesome.


Reply to this email directly or view it on GitHubhttps://github.com/getpelican/pelican/pull/618#issuecomment-12128911.

Contributor

bbinet commented Jan 11, 2013

I won't review this one because I'm not confortable with py3k yet, I need
to learn about py2k/py3k differences first.
But I'm +1 to merge this one soon so that other pull requests can be
reviewed.

On 11 January 2013 03:08, Alexis Metaireau notifications@github.com wrote:

I pushed a working version, from the work by @dmdmhttps://github.com/dmdmat
https://github.com/getpelican/pelican/commits/py3k

which is basically this PR with only one commit for the code: b953900b953900 one for the docs
4d4efc64d4efc6

The last thing to do is to add back these tests:
http://paste.lolnet.org/mirawugolu.py, ported to unified py2 & 3, and
then we're okay to merge!

@justinmayer https://github.com/justinmayer @bbinethttps://github.com/bbinet
@tbunnyman https://github.com/tbunnyman if one of you can have a look
at these commits and check that they look okay, that would be awesome.


Reply to this email directly or view it on GitHubhttps://github.com/getpelican/pelican/pull/618#issuecomment-12128911.

@hukka

This comment has been minimized.

Show comment Hide comment
@hukka

hukka Jan 11, 2013

I left a few comments, seems like it doesn't update this issue.

hukka commented Jan 11, 2013

I left a few comments, seems like it doesn't update this issue.

@almet

This comment has been minimized.

Show comment Hide comment
@almet

almet Jan 15, 2013

Owner

71995d5 takes care of py3k. Thanks all.

Owner

almet commented Jan 15, 2013

71995d5 takes care of py3k. Thanks all.

@almet almet closed this Jan 15, 2013

@justinmayer justinmayer referenced this pull request Jan 16, 2013

Closed

Py3k v2 #495

This was referenced Feb 10, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment