Fix bdist_rpm by setting default encoding in setup.py

[TauPan]

I've tried to create a rpm via setup.py bdist_rpm and stumbled accross the following error:

running bdist_rpm                                            
running egg_info
writing cpe.egg-info/PKG-INFO
writing top-level names to cpe.egg-info/top_level.txt
writing dependency_links to cpe.egg-info/dependency_links.txt
reading manifest file 'cpe.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'cpe/tests'
writing manifest file 'cpe.egg-info/SOURCES.txt'
writing 'build/bdist.linux-x86_64/rpm/SPECS/cpe.spec'
Traceback (most recent call last):
  File "setup.py", line 31, in <module>
    zip_safe=False,
  File "/home/taupan/.pythonbrew/pythons/Python-2.6/lib/python2.6/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/home/taupan/.pythonbrew/pythons/Python-2.6/lib/python2.6/distutils/dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "/home/taupan/.pythonbrew/pythons/Python-2.6/lib/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
  File "/home/taupan/.virtualenvs/airshield-cve/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/bdist_rpm.py", line 28, in run
  File "/home/taupan/.pythonbrew/pythons/Python-2.6/lib/python2.6/distutils/command/bdist_rpm.py", line 293, in run
    "writing '%s'" % spec_path)
  File "/home/taupan/.pythonbrew/pythons/Python-2.6/lib/python2.6/distutils/cmd.py", line 358, in execute
    util.execute(func, args, msg, dry_run=self.dry_run)
  File "/home/taupan/.pythonbrew/pythons/Python-2.6/lib/python2.6/distutils/util.py", line 373, in execute
    apply(func, args)
  File "/home/taupan/.pythonbrew/pythons/Python-2.6/lib/python2.6/distutils/file_util.py", line 252, in write_file
    f.write(line + "\n")
UnicodeEncodeError: 'ascii' codec can't encode character u'\xed' in position 30: ordinal not in range(128)

I fixed it by setting the default encoding in setup.py, see http://stackoverflow.com/a/1178429

I'm not sure if this is a good fix.

[nilp0inter]

Hi! @TauPan,

Thank you for your pull request.

It seems that distutils is trying to write a file without previously setting any encoding so python uses the default encoding (in python 2 is ascii). This make program crash because we have non ascii characters in some files. (Author names for example 👯 )

On one hand the patch you provided is a common, but very discouraged, hack in Python 2.x. On the other hand this only apply to the setup.py... so I think it's OK.

However we are improving the CPE module compatibility against python versions and port to 3 right now. This hack is a no-op in python 3 (and UTF-8 is the default encoding now) so we need to set this only for python 2. http://stackoverflow.com/a/11742928

In conclusion, I think we'll take this workaround for python 2 (if don't break anything) but we should wait until the new changes and prior to merge.

What do you think @galindale?

[TauPan]

Hi @nilp0inter!

I have to disclose that I'm pretty new to python development (started developing python in may 2013 on a regular basis, although I started experiments in python a decade ago or so), so I don't have a good feel for what is "pythonic" and what isn't, yet. Also I did very little research before issuing the pull request, which is why commented that I'm "not sure" about it.

While that stackoverflow post is informative, I ask myself what would be the recommended way to solve this particular problem?

I can't help suspecting that my proposal is a viable 80% solution.

One last question: will you still support python 2 after porting to 3? I have some ideas for useful features (such as modelling the behaviour of cpe sets more closely to python sets), and I don't know how hard/easy it is to support both major python versions. I only coded python 2 so far. We'll need python 2 support for our project (which I'm not at liberty to disclose here, yet).

Kind regards
Friedel

[TauPan]

I'm afraid this ticket is invalid. I made a blunder with pythonbrew and was using python version 2.6 (sic!) which has some encoding issues.

This completely goes away with python 2.6.6.

I think this can be closed.

[nilp0inter]

Hi @TauPan,

I was trying to reproduce the error today. I sucess on CentOS 5 with python 2.4 but I can't with CentOS 6 and python 2.6. I agree with you and I'm going to close the issue.

Just for the record, I found this useful information:

• Issue 2562: Cannot use non-ascii letters in disutils if setuptools is used. - Python tracker
• cpython: 4c683ec4415b
• bdist_rpm is dead, long life to Py2RPM
• Issue 9561: distutils: set encoding to utf-8 for input and output files - Python tracker
• openSUSE:Specfile guidelines - openSUSE

Answering your first question "would be the recommended way to solve this particular problem?" I was thinking on an approach similar to the django underscore function so in older versions the text would be encoded to utf-8 and in newer versions it remains unicode. Something like this https://gist.github.com/nilp0inter/7900502

And in reply to your last question about the compatibility: Yes. We'll remain compatibility with 2.x as our projects are currently 2.7. So it's a must.

I really like to hear your ideas for features and improvements. Could you please open a separated issue for discuss that?