Permalink
Browse files

Port pelican to python 3.

Stays compatible with 2.x series, thanks to an unified codebase.
  • Loading branch information...
1 parent 9847394 commit 71995d5e1bec6f34ef836e1dc7416e27dd3cfe30 Dirk Makowski committed with almet Jan 11, 2013
View
@@ -11,3 +11,5 @@ tags
.tox
.coverage
htmlcov
+six-*.egg/
+*.orig
View
@@ -2,11 +2,13 @@ language: python
python:
- "2.6"
- "2.7"
+# - "3.2"
before_install:
@hukka

hukka Jan 11, 2013

Why 3.2, and not 3.3?

@almet

almet Jan 11, 2013

Owner

let's start with 3.2 we'll see for 3.3 later? :)

@dmdm

dmdm Jan 11, 2013

Contributor

We should remove 2.6 here.

@justinmayer

justinmayer Jan 11, 2013

Owner

I'm with @tpievila. What's the argument for the focus on 3.2 instead of 3.3?

@onlyhavecans

onlyhavecans Jan 11, 2013

Owner

Are there majot changes in 3.2->3.3 that really affects the way pelican runs?

@almet

almet Jan 11, 2013

Owner

I'm not saying that we shouldn't support 3.3, but if we can merge it for 3.2 only for now, that would be okay, we can check support for 3.3 later on; we waited too much for this already, I think.

@almet

almet Jan 11, 2013

Owner

@tbunnyman I don't know. Maybe. Maybe not, we need to check this out which means more work. Not sure we want that for now.

@onlyhavecans

onlyhavecans Jan 11, 2013

Owner

I can get behind that @ametaireau

- sudo apt-get update -qq
- sudo apt-get install -qq ruby-sass
install:
- - pip install nose unittest2 mock --use-mirrors
+ - pip install nose mock --use-mirrors
+ - if [[ $TRAVIS_PYTHON_VERSION == '3.2' ]]; then pip install --use-mirrors unittest2py3k; else pip install --use-mirrors unittest2; fi
- pip install . --use-mirrors
- pip install Markdown
- pip install webassets
View
@@ -1,8 +1,9 @@
# Tests
-unittest2
mock
+
@onlyhavecans

onlyhavecans Jan 11, 2013

Owner

six should be in here I beleve

@almet

almet Jan 11, 2013

Owner

nope, it's a hard dep of pelican so you don't need here. Same applies for BeautifulSoup4 and lxml I think, actually.

# Optional Packages
Markdown
-BeautifulSoup
+BeautifulSoup4
+lxml
typogrify
-webassets
+webassets
View
@@ -4,7 +4,7 @@ Release history
3.2 (XXXX-XX-XX)
================
-* [...]
+* Support for Python 3!
3.1 (2012-12-04)
================
View
@@ -1,4 +1,5 @@
# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
import sys, os
sys.path.append(os.path.abspath('..'))
@@ -10,8 +11,8 @@
extensions = ['sphinx.ext.autodoc',]
source_suffix = '.rst'
master_doc = 'index'
-project = u'Pelican'
-copyright = u'2010, Alexis Metaireau and contributors'
+project = 'Pelican'
+copyright = '2010, Alexis Metaireau and contributors'
exclude_patterns = ['_build']
version = __version__
release = __major__
@@ -34,16 +35,16 @@
# -- Options for LaTeX output --------------------------------------------------
latex_documents = [
- ('index', 'Pelican.tex', u'Pelican Documentation',
- u'Alexis Métaireau', 'manual'),
+ ('index', 'Pelican.tex', 'Pelican Documentation',
+ 'Alexis Métaireau', 'manual'),
]
# -- Options for manual page output --------------------------------------------
man_pages = [
- ('index', 'pelican', u'pelican documentation',
- [u'Alexis Métaireau'], 1),
- ('pelican-themes', 'pelican-themes', u'A theme manager for Pelican',
- [u'Mickaël Raybaud'], 1),
- ('themes', 'pelican-theming', u'How to create themes for Pelican',
- [u'The Pelican contributors'], 1)
+ ('index', 'pelican', 'pelican documentation',
+ ['Alexis Métaireau'], 1),
+ ('pelican-themes', 'pelican-themes', 'A theme manager for Pelican',
+ ['Mickaël Raybaud'], 1),
+ ('themes', 'pelican-theming', 'How to create themes for Pelican',
+ ['The Pelican contributors'], 1)
]
View
@@ -1,3 +1,7 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals, print_function
+import six
+
import os
import re
import sys
@@ -55,7 +59,7 @@ def init_plugins(self):
self.plugins = self.settings['PLUGINS']
for plugin in self.plugins:
# if it's a string, then import it
- if isinstance(plugin, basestring):
+ if isinstance(plugin, six.string_types):
logger.debug("Loading plugin `{0}' ...".format(plugin))
plugin = __import__(plugin, globals(), locals(), 'module')
@@ -265,7 +269,7 @@ def get_instance(args):
settings = read_settings(args.settings, override=get_config(args))
cls = settings.get('PELICAN_CLASS')
- if isinstance(cls, basestring):
+ if isinstance(cls, six.string_types):
module, cls_name = cls.rsplit('.', 1)
module = __import__(module)
cls = getattr(module, cls_name)
@@ -311,15 +315,15 @@ def main():
"Nothing to generate.")
files_found_error = False
time.sleep(1) # sleep to avoid cpu load
- except Exception, e:
+ except Exception as e:
logger.warning(
"Caught exception \"{}\". Reloading.".format(e)
)
continue
else:
pelican.run()
- except Exception, e:
- logger.critical(unicode(e))
+ except Exception as e:
+ logger.critical(e)
if (args.verbosity == logging.DEBUG):
raise
View
@@ -1,4 +1,7 @@
# -*- coding: utf-8 -*-
+from __future__ import unicode_literals, print_function
+import six
+
import copy
import locale
import logging
@@ -11,8 +14,10 @@
from pelican.settings import _DEFAULT_CONFIG
-from pelican.utils import slugify, truncate_html_words, memoized
+from pelican.utils import (slugify, truncate_html_words, memoized,
+ python_2_unicode_compatible)
from pelican import signals
+import pelican.utils
logger = logging.getLogger(__name__)
@@ -85,13 +90,8 @@ def __init__(self, content, metadata=None, settings=None,
self.date_format = self.date_format[1]
if hasattr(self, 'date'):
- encoded_date = self.date.strftime(
- self.date_format.encode('ascii', 'xmlcharrefreplace'))
-
- if platform == 'win32':
- self.locale_date = encoded_date.decode(stdin.encoding)
- else:
- self.locale_date = encoded_date.decode('utf')
+ self.locale_date = pelican.utils.strftime(self.date,
+ self.date_format)
# manage status
if not hasattr(self, 'status'):
@@ -167,7 +167,7 @@ def replacer(m):
origin = '/'.join((siteurl,
self._context['filenames'][value].url))
else:
- logger.warning(u"Unable to find {fn}, skipping url"
+ logger.warning("Unable to find {fn}, skipping url"
" replacement".format(fn=value))
return m.group('markup') + m.group('quote') + origin \
@@ -243,10 +243,10 @@ class Article(Page):
class Quote(Page):
base_properties = ('author', 'date')
-
+@python_2_unicode_compatible
class URLWrapper(object):
def __init__(self, name, settings):
- self.name = unicode(name)
+ self.name = name
self.slug = slugify(self.name)
self.settings = settings
@@ -257,12 +257,9 @@ def __hash__(self):
return hash(self.name)
def __eq__(self, other):
- return self.name == unicode(other)
+ return self.name == other
def __str__(self):
- return str(self.name.encode('utf-8', 'replace'))
-
- def __unicode__(self):
return self.name
def _from_settings(self, key, get_page_name=False):
@@ -272,14 +269,14 @@ def _from_settings(self, key, get_page_name=False):
Useful for pagination."""
setting = "%s_%s" % (self.__class__.__name__.upper(), key)
value = self.settings[setting]
- if not isinstance(value, basestring):
- logger.warning(u'%s is set to %s' % (setting, value))
+ if not isinstance(value, six.string_types):
+ logger.warning('%s is set to %s' % (setting, value))
return value
else:
if get_page_name:
- return unicode(os.path.splitext(value)[0]).format(**self.as_dict())
+ return os.path.splitext(value)[0].format(**self.as_dict())
else:
- return unicode(value).format(**self.as_dict())
+ return value.format(**self.as_dict())
page_name = property(functools.partial(_from_settings, key='URL', get_page_name=True))
url = property(functools.partial(_from_settings, key='URL'))
@@ -292,13 +289,14 @@ class Category(URLWrapper):
class Tag(URLWrapper):
def __init__(self, name, *args, **kwargs):
- super(Tag, self).__init__(unicode.strip(name), *args, **kwargs)
+ super(Tag, self).__init__(name.strip(), *args, **kwargs)
class Author(URLWrapper):
pass
+@python_2_unicode_compatible
class StaticContent(object):
def __init__(self, src, dst=None, settings=None):
if not settings:
@@ -309,17 +307,14 @@ def __init__(self, src, dst=None, settings=None):
self.save_as = os.path.join(settings['OUTPUT_PATH'], self.url)
def __str__(self):
- return str(self.filepath.encode('utf-8', 'replace'))
-
- def __unicode__(self):
return self.filepath
def is_valid_content(content, f):
try:
content.check_properties()
return True
- except NameError, e:
- logger.error(u"Skipping %s: impossible to find informations about"
+ except NameError as e:
+ logger.error("Skipping %s: impossible to find informations about"
"'%s'" % (f, e))
return False
View
@@ -1,10 +1,11 @@
# -*- coding: utf-8 -*-
+from __future__ import unicode_literals, print_function
+
import os
import math
import random
import logging
import datetime
-import subprocess
import shutil
from codecs import open
@@ -119,7 +120,7 @@ def _update_context(self, items):
for item in items:
value = getattr(self, item)
if hasattr(value, 'items'):
- value = value.items()
+ value = list(value.items())
@hukka

hukka Jan 11, 2013

Might be a good idea to mark this and other uses of list() as explicit Python 3 safeguard against iterators.

self.context[item] = value
@@ -133,8 +134,8 @@ def get_source(self, environment, template):
if template != self.path or not os.path.exists(self.fullpath):
raise TemplateNotFound(template)
mtime = os.path.getmtime(self.fullpath)
- with file(self.fullpath) as f:
- source = f.read().decode('utf-8')
+ with open(self.fullpath, 'r', encoding='utf-8') as f:
+ source = f.read()
return source, self.fullpath, \
lambda: mtime == os.path.getmtime(self.fullpath)
@@ -323,8 +324,8 @@ def generate_context(self):
try:
signals.article_generate_preread.send(self)
content, metadata = read_file(f, settings=self.settings)
- except Exception, e:
- logger.warning(u'Could not process %s\n%s' % (f, str(e)))
+ except Exception as e:
+ logger.warning('Could not process %s\n%s' % (f, str(e)))
continue
# if no category is set, use the name of the path as a category
@@ -333,8 +334,7 @@ def generate_context(self):
if (self.settings['USE_FOLDER_AS_CATEGORY']
and os.path.dirname(f) != article_path):
# if the article is in a subdirectory
- category = os.path.basename(os.path.dirname(f))\
- .decode('utf-8')
+ category = os.path.basename(os.path.dirname(f))
else:
# if the article is not in a subdirectory
category = self.settings['DEFAULT_CATEGORY']
@@ -366,8 +366,8 @@ def generate_context(self):
elif article.status == "draft":
self.drafts.append(article)
else:
- logger.warning(u"Unknown status %s for file %s, skipping it." %
- (repr(unicode.encode(article.status, 'utf-8')),
+ logger.warning("Unknown status %s for file %s, skipping it." %
+ (repr(article.status),
repr(f)))
self.articles, self.translations = process_translations(all_articles)
@@ -394,7 +394,7 @@ def generate_context(self):
tag_cloud = sorted(tag_cloud.items(), key=itemgetter(1), reverse=True)
tag_cloud = tag_cloud[:self.settings.get('TAG_CLOUD_MAX_ITEMS')]
- tags = map(itemgetter(1), tag_cloud)
+ tags = list(map(itemgetter(1), tag_cloud))
if tags:
max_count = max(tags)
steps = self.settings.get('TAG_CLOUD_STEPS')
@@ -450,8 +450,8 @@ def generate_context(self):
exclude=self.settings['PAGE_EXCLUDES']):
try:
content, metadata = read_file(f, settings=self.settings)
- except Exception, e:
- logger.warning(u'Could not process %s\n%s' % (f, str(e)))
+ except Exception as e:
+ logger.warning('Could not process %s\n%s' % (f, str(e)))
continue
signals.pages_generate_context.send(self, metadata=metadata)
page = Page(content, metadata, settings=self.settings,
@@ -466,8 +466,8 @@ def generate_context(self):
elif page.status == "hidden":
hidden_pages.append(page)
else:
- logger.warning(u"Unknown status %s for file %s, skipping it." %
- (repr(unicode.encode(page.status, 'utf-8')),
+ logger.warning("Unknown status %s for file %s, skipping it." %
+ (repr(page.status),
repr(f)))
self.pages, self.translations = process_translations(all_pages)
@@ -550,15 +550,15 @@ def _create_pdf(self, obj, output_path):
# print "Generating pdf for", obj.filename, " in ", output_pdf
with open(obj.filename) as f:
self.pdfcreator.createPdf(text=f.read(), output=output_pdf)
- logger.info(u' [ok] writing %s' % output_pdf)
+ logger.info(' [ok] writing %s' % output_pdf)
def generate_context(self):
pass
def generate_output(self, writer=None):
# we don't use the writer passed as argument here
# since we write our own files
- logger.info(u' Generating PDF files...')
+ logger.info(' Generating PDF files...')
pdf_path = os.path.join(self.output_path, 'pdf')
if not os.path.exists(pdf_path):
try:
@@ -583,6 +583,6 @@ def _create_source(self, obj, output_path):
copy('', obj.filename, dest)
def generate_output(self, writer=None):
- logger.info(u' Generating source files...')
+ logger.info(' Generating source files...')
for object in chain(self.context['articles'], self.context['pages']):
self._create_source(object, self.output_path)
Oops, something went wrong.

13 comments on commit 71995d5

It was pretty big patch, and I think my concentration started to falter in the end. But here's a few questions and suggestions.

Owner

almet replied Jan 11, 2013

thanks; @dmdm, what do you think? all remarks sounds reasonable to me, I'm willing to update the patch with these remarks.

Contributor

dmdm replied Jan 11, 2013

Wrote some comments on the comments. As a whole, it looks good for me.

Owner

onlyhavecans replied Jan 11, 2013

This is maybe just a little thing but;
With this test, in order to keep compatibility our new official testing method is tox yes? The contribution docs should really be updated to go over the new testing and development procedure. It might also want to note a lot of the things save in the INI in the actual docs

Owner

onlyhavecans replied Jan 11, 2013

Can anyone verify this failure?
Mac 10.8.2 Python 2.7.2, fresh virtenv, fresh checkout

1197e09 Fixes this

Owner

almet replied Jan 11, 2013

These should be fixed, the patch was removing some previous changes, see 1197e09

I also adressed @tpievila comments in 149ca49 and removed support for python2.6 in 4ac0949

Tests are failing here, however, and I don't get why. Can anyone test?

Contributor

dmdm replied Jan 11, 2013

Checked out this py3k branch on a fresh Python 3.2 venv, I got these results with "unit2 discover":

  1. The tests did not run at all. Obviously, unit2 failed to import the test cases. Renaming the directory "tests" to sth else did not solve the problem, but deactivating and reentering the venv did -- which is a derivation of a well-known procedure. Why this had to be applied here eludes me.
  2. The importer-test fails, like tbunnyman had described earlier. If this test ever succeeded I do not know, it is skipped automatically when pandoc is not installed, which was the case here until recently.
Contributor

dmdm replied Jan 11, 2013

About Py33: There are (imho) severe differences between 3.3 and 3.2, e.g. in the exception hierarchy, but afaihe*) these changes are backwards compatible. I.o.w., Pelican should run unchaged in 3.3 as well. However, at least for Linux, the creation of a venv with virtualenvwrapper, as is kind of a standard here, does not (yet?) support 3.3. If, and how, Pelican runs in a venv that was created with 3.3's new on-board tools has to be tested.

*) as far as I had experienced

Owner

onlyhavecans replied Jan 11, 2013

Mac Py3.3 note....
Since using the python.org packages bring a bad time you need to use homebrew... which is a bit bleeding edge so it's what me and likely @justinmayer will be doing all our testing on anyways.

I've had no problems with venv with py 3.3 but I've not done serious testing

Owner

justinmayer replied Jan 11, 2013

I don't think there's much that's bleeding edge about Homebrewed Python on Mac OS X. It's actually become quite standard among professional Python developers, or at least so I'm told.

Owner

almet replied Jan 13, 2013

That should be working now. Is it?

Owner

almet replied Jan 13, 2013

I propose we merge this by tomorrow. @justinmayer tell me if you find anything weird and if you're able to run everything smoothly, if you have time :-) all is working for me here.

Owner

justinmayer replied Jan 13, 2013

Results of my py3k branch testing: No unit2 test errors on freshly-created virtualenvs, one each for both Python 3.3.0 and 2.7.3. Similarly, generation of sample content proceeded without errors. I think we are all set to go!

Please sign in to comment.