Add support for using Zulip with Python 3 #256

Open
timabbott opened this Issue Nov 4, 2015 · 60 comments

Projects

None yet

3 participants

@timabbott
Member

Python 3 enables a bunch of awesome stuff like optional static typing, and I think is actually pretty feasible to add. I've already applied the fast majority of the futurize/modernize fixers to the repository in a way that preserves Python 2+3 support; for future work, we should continue using that approach. Based on some hackish testing, we're about 100 LOC away from passing 90% of backend tests against Python 3 (well not including the #! lines that hardcode Python 2.7). Based on that investigation, the main projects that remain are as follows.

We have some dependencies that don't support Python 3:

  • python-twitter (https://github.com/bear/python-twitter). It looks like some work has been done for Python 3 support there (bear/python-twitter#155) or we could switch to an alternative library; Zulip primarily uses it to render links to Tweets.
  • python-ldap and django-auth-ldap. It appears there are Python 3 supporting libraries python-ldap3 and django-python3-ldap but they seem to have a different interface; so the challenges are (1) decide whether we want to switch to the python2+3 versions and (2) if so, actually do it.
  • Old django-pipeline. We're using an old version but a newer version that supports Python 3 exists; this upgrade may be somewhat messy since it involves renaming things like "load compressed" and "compressed_css" in our Django templates.

(We can update this list to add others but those are the main ones where we need to do more than a straightforward version upgrade).

There are also some changes needed to the Zulip code itself still:

  • Doing a few more Python 3 compatibility migrations (I've already done like 20 of them), in particular the file=>open change, .items() iteration, and several libraries like StringIO, urlparse. For these, we can fix things and then add the fixers to the tests in #252 to avoid regressions.
  • Unicode transition fixes such as adding appropriate encode/decode usage where we use base-64 encoding, hash digests, etc.

If anyone's interested in working on Python 3 support, feel free to claim one of these pieces on this issue's discussion thread!

I pushed https://github.com/timabbott/zulip/commits/tabbott-py3k-testing which has some total hack commits I wrote as part of the testing process for Python 3 support; with some library version installation work that branch passes 90% of our tests. That could be helpful for anyone working on this.

@timabbott timabbott referenced this issue in bear/python-twitter Nov 4, 2015
Closed

Installation error with python 3.4.0 #155

@foxmask
foxmask commented Nov 4, 2015

About twitter i switched to twython which is great

@sharmaeklavya2
Contributor

I would like to start with Unicode transition fixes. I have worked with base64 encode/decode in python 3 in the past. Where and how should I begin?

@timabbott
Member

@sharmaeklavya2 I just merged #443 into master, which addresses the issues I'd tracked down already. I think the best place to start working on the python 3 support would be to build a python3 venv from the Zulip requirements.txt and run the Zulip backend test suite using python 3. As mentioned above, I have some hackish work from before on this branch:
https://github.com/timabbott/zulip/commits/tabbott-py3k-testing which may be useful for helping you identify the requirements.txt changes needed and what code you need to comment out to disable the LDAP integration (so that you can work on other issues). Much of the other stuff in that branch has since been cleaned up and merged into master.

The python-twitter folks merged python 3 support, so that part may just require updating the version in requirements.txt. Let me know if you get stuck and need any help!

@sharmaeklavya2
Contributor

I tried making a python3 venv a few hours ago using requirements.txt. That failed for some reason. I'll get back with the error messages.

I should work on zulip's current master branch and use your branch only for reference, right? Or should I work on your branch?

@timabbott
Member

I'd recommend working on top of master and using that branch for reference -- it's pretty hacky and most of the valuable stuff has been merged; the main things I'd expect to get out of that old branch is how to comment out LDAP and a maybe subset of the requirements.txt updates you'll need to make.

@sharmaeklavya2
Contributor

The package enum34==1.0.4 is causing problems while pip install -r requirements.txt in the python3 virtualenv.

Downloading/unpacking enum34==1.0.4
  Downloading enum34-1.0.4.tar.gz
  Running setup.py (path:/tmp/pip-build-kcrl4cod/enum34/setup.py) egg_info for package enum34
    Traceback (most recent call last):
      File "/tmp/pip-build-kcrl4cod/enum34/enum/__init__.py", line 371, in __getattr__
        return cls._member_map_[name]
    KeyError: '_convert'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 3, in <module>
      File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/setuptools/__init__.py", line 12, in <module>
        from setuptools.extension import Extension
      File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/setuptools/extension.py", line 8, in <module>
        from .dist import _get_unpatched
      File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/setuptools/dist.py", line 16, in <module>
        from setuptools.depends import Require
      File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/setuptools/depends.py", line 6, in <module>
        from setuptools import compat
      File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/setuptools/compat.py", line 45, in <module>
        import http.client as httplib
      File "/usr/lib/python3.4/http/client.py", line 69, in <module>
        import email.parser
      File "/usr/lib/python3.4/email/parser.py", line 12, in <module>
        from email.feedparser import FeedParser, BytesFeedParser
      File "/usr/lib/python3.4/email/feedparser.py", line 27, in <module>
        from email import message
      File "/usr/lib/python3.4/email/message.py", line 16, in <module>
        from email import utils
      File "/usr/lib/python3.4/email/utils.py", line 29, in <module>
        import socket
      File "/usr/lib/python3.4/socket.py", line 72, in <module>
        IntEnum._convert(
      File "/tmp/pip-build-kcrl4cod/enum34/enum/__init__.py", line 373, in __getattr__
        raise AttributeError(name)
    AttributeError: _convert
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "/tmp/pip-build-kcrl4cod/enum34/enum/__init__.py", line 371, in __getattr__

    return cls._member_map_[name]

KeyError: '_convert'



During handling of the above exception, another exception occurred:



Traceback (most recent call last):

  File "<string>", line 3, in <module>

  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/setuptools/__init__.py", line 12, in <module>

    from setuptools.extension import Extension

  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/setuptools/extension.py", line 8, in <module>

    from .dist import _get_unpatched

  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/setuptools/dist.py", line 16, in <module>

    from setuptools.depends import Require

  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/setuptools/depends.py", line 6, in <module>

    from setuptools import compat

  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/setuptools/compat.py", line 45, in <module>

    import http.client as httplib

  File "/usr/lib/python3.4/http/client.py", line 69, in <module>

    import email.parser

  File "/usr/lib/python3.4/email/parser.py", line 12, in <module>

    from email.feedparser import FeedParser, BytesFeedParser

  File "/usr/lib/python3.4/email/feedparser.py", line 27, in <module>

    from email import message

  File "/usr/lib/python3.4/email/message.py", line 16, in <module>

    from email import utils

  File "/usr/lib/python3.4/email/utils.py", line 29, in <module>

    import socket

  File "/usr/lib/python3.4/socket.py", line 72, in <module>

    IntEnum._convert(

  File "/tmp/pip-build-kcrl4cod/enum34/enum/__init__.py", line 373, in __getattr__

    raise AttributeError(name)

AttributeError: _convert

----------------------------------------
Cleaning up...
@sharmaeklavya2
Contributor

I just now read up what enum34 does. It's probably not needed on python 3.4. I'll just remove it and see what happens.

@sharmaeklavya2
Contributor

I ran caniusepython3 and it said:

You need 6 projects to transition to Python 3.
Of those 6 projects, 6 have no direct dependencies blocking their transition:

  apns-client
  fonttools
  gcm-client
  mandrill
  pydns
  python-ldap

After removing these packages from requirements.txt, I got an error while installing wsgiref==0.1.2. Since that comes with newer versions of python, it's probably not needed.

@timabbott
Member

If I recall correctly, several of those projects actually do have a python 3 supporting version available -- if you check them out on github, you can usually find things like a .travis.yml or setup.py showing whether they actually do support python 3 with a newer version, or in other cases that the package was renamed when python 3 support was added.

(And yeah, some packages have been removed in the latest python 3...)

@sharmaeklavya2
Contributor

Yeah I found that py3dns is the python3 port of pydns. I'll similarly check for other packages.

I also got a lot of warnings while pip install.

@sharmaeklavya2
Contributor

I anyways ran ./tools/test-all and I got a lot of errors in ./tools/lint-all.

api/integrations/basecamp/zulip_basecamp_mirror:173: undefined name 'basestring'
assets/favicon/generate:13: undefined name 'xrange'
bots/process_ccache:17: undefined name 'file'
bots/process_ccache:22: undefined name 'file'
bots/process_ccache:28: undefined name 'file'
bots/process_ccache:32: undefined name 'file'
bots/sync-public-streams:62: undefined name 'file'
provision.py:151: undefined name 'execfile'
tools/emoji_dump/emoji_dump.py:25: undefined name 'unichr'
zerver/lib/actions.py:2332: undefined name 'unichr'
api/examples/create-user:49:12: invalid syntax
print client.create_user({
           ^
api/examples/edit-message:56:12: invalid syntax
print client.update_message(message_data)
           ^
api/examples/get-public-streams:46:12: invalid syntax
print client.get_streams(include_public=True, include_subscribed=False)
           ^
api/examples/list-members:45:14: invalid syntax
    print user["full_name"], user["email"]
             ^
api/examples/list-subscriptions:45:12: invalid syntax
print client.list_subscriptions()
           ^
api/examples/print-events:46:15: invalid syntax
    print event
              ^
api/examples/print-messages:46:17: invalid syntax
    print message
                ^
api/examples/print-next-message:45:12: invalid syntax
print client.get_messages({})
           ^
api/examples/send-message:57:12: invalid syntax
print client.send_message(message_data)
           ^
api/examples/subscribe:51:12: invalid syntax
print client.add_subscriptions([{"name": stream_name} for stream_name in
           ^
api/examples/unsubscribe:51:12: invalid syntax
print client.remove_subscriptions(options.streams.split())
           ^
api/integrations/asana/zulip_asana_mirror:47:19: invalid syntax
except ImportError, e:
                  ^
api/integrations/codebase/zulip_codebase_mirror:43:19: invalid syntax
except ImportError, e:
                  ^
api/integrations/rss/rss-bot:90:19: invalid syntax
    except OSError, e:
                  ^
api/integrations/twitter/twitter-bot:115:100: invalid syntax
    print "Unable to log in to twitter with supplied credentials. Please double-check and try again"
                                                                                                   ^
api/integrations/twitter/twitter-search-bot:142:35: invalid syntax
Please double-check and try again."
                                  ^
bin/get-django-setting:11:13: invalid syntax
print getattr(settings, sys.argv[1])
            ^
bots/check-mirroring:103:16: invalid syntax
    print status
               ^
bots/check-rabbitmq-consumers:18:64: invalid syntax
    print "This script must be run as the root or rabbitmq user"
                                                               ^
bots/check-rabbitmq-queue:33:64: invalid syntax
    print "This script must be run as the root or rabbitmq user"
                                                               ^
bots/gcal-bot:102:41: invalid syntax
                print 'Sending reminder:', line
                                        ^
bots/log2zulip:23:19: invalid syntax
    except OSError, e:
                  ^
frontend_tests/run-casper:53:27: invalid syntax
        raise RuntimeError, 'Server died unexpectedly! Check frontend_tests/casper_tests/server.log'
                          ^
puppet/zulip/files/postgresql/process_fts_updates:47:19: invalid syntax
except ImportError, e:
                  ^
puppet/zulip_internal/files/nagios_plugins/check_fts_update_log:17:47: invalid syntax
    print "%s: %s rows in fts_update_log table" % (state, num)
                                              ^
puppet/zulip_internal/files/nagios_plugins/check_personal_zephyr_mirrors:25:18: invalid syntax
    print "%s\n%s" % (state, output)
                 ^
puppet/zulip_internal/files/nagios_plugins/check_pg_replication_lag:19:18: invalid syntax
    print "%s: %s" % (state, msg)
                 ^
puppet/zulip_internal/files/nagios_plugins/check_postgres_backup:16:18: invalid syntax
    print "%s: %s" % (state, msg)
                 ^
puppet/zulip_internal/files/nagios_plugins/check_rabbitmq_consumers:18:62: invalid syntax
    print "Please pass the name of the consumer file to check"
                                                             ^
puppet/zulip_internal/files/nagios_plugins/check_rabbitmq_queues:21:12: invalid syntax
print result
           ^
puppet/zulip_internal/files/nagios_plugins/check_send_receive_time:45:75: invalid syntax
    print 'No output options specified! Please provide --munin or --nagios'
                                                                          ^
puppet/zulip_internal/files/nagios_plugins/check_user_zephyr_mirror_liveness:40:20: invalid syntax
    print "%s: %s%s" % (state, short_msg, too_old_data)
                   ^
puppet/zulip_internal/files/nagios_plugins/check_zephyr_mirror:25:49: invalid syntax
    print "%s: Last test run completed at %s\n%s" % (
                                                ^
scripts/lib/unpack-zulip:15:14: invalid syntax
    print FAIL + "Usage: %s <tarball>" % (sys.argv[0],) + ENDC
             ^
scripts/lib/upgrade-zulip:23:14: invalid syntax
    print FAIL + "Usage: %s <tarball>" % (sys.argv[0],) + ENDC
             ^
scripts/lib/upgrade-zulip-stage-2:25:14: invalid syntax
    print FAIL + "Usage: %s <deploy path>" % (sys.argv[0],) + ENDC
             ^
scripts/restart-server:45:13: invalid syntax
print OKGREEN + "Application restarted successfully!" + ENDC
            ^
tools/compile-handlebars-templates:39:41: invalid syntax
            print 'Recompiling templates'
                                        ^
tools/deprecated/inject-messages/inject-messages:72:48: invalid syntax
        print "Could not send message (API Key " + characters[character]["api-key"] + "):"
                                               ^
tools/deprecated/review:182:50: invalid syntax
        print "Attempting to push %s to remote %s" % (local_name, remote)
                                                 ^
tools/get-handlebar-vars:101:19: invalid syntax
        print '===', fn
                  ^
tools/lint-all:130:30: invalid syntax
                    print line
                             ^
tools/minify-js:51:77: invalid syntax
        print "Warning: git returned an error when comparing to the previous"
                                                                            ^
tools/post-receive:77:26: invalid syntax
                print FAIL + "Schema change detected!  Please make the appropriate changes manually." + ENDC
                         ^
tools/print-all/print-all:118:20: invalid syntax
    print 'Scanning', fil
                   ^
tools/python-proxy:118:25: invalid syntax
        print 'self.path', self.path
                        ^
tools/test-js-with-casper:53:27: invalid syntax
        raise RuntimeError, 'Server died unexpectedly! Check frontend_tests/casper_tests/server.log'
                          ^
tools/update-deployment:17:14: invalid syntax
    print FAIL + "Usage: update-deployment refname" + ENDC
             ^
tools/zulip-export/zulip-export:59:28: invalid syntax
print "Fetching messages..."
                           ^
zulip_tools.py:40:19: invalid syntax
    except OSError, e:
                  ^
Warning: The use of 'import' is deprecated at /mnt/Code/git_repos/forked/zulip/puppet/puppet-common/tests/concatfilepart1.pp:3. See http://links.puppetlabs.com/puppet-import-deprecation
   (at /usr/lib/ruby/vendor_ruby/puppet/parser/parser_support.rb:110:in `import')

FAILED ./tools/lint-all

You can see that most of them are about print statements, except, raise, file, etc. These should be easy to fix IMO.

By the way, is it okay to post such large outputs here?

@timabbott
Member
@sharmaeklavya2
Contributor

I realized that scripts start with #!/usr/bin/env python2.7. I thought that I was running them with a python3 venv but that's not the case. Doing python tools/lint-all gives this output:

  File "./tools/lint-all", line 130
    print line
             ^
SyntaxError: Missing parentheses in call to 'print'
@timabbott
Member

Yeah probably you'll just want to replace that hardcoded python2.7 with
"python" for this purpose :)

    -Tim Abbott

On Wed, Mar 9, 2016 at 11:09 AM, Eklavya Sharma notifications@github.com
wrote:

I realized that scripts start with #!/usr/bin/env python2.7. I thought
that I was running them with a python3 venv but that's not the case. Doing python
tools/lint-all gives this output:

File "./tools/lint-all", line 130
print line
^
SyntaxError: Missing parentheses in call to 'print'


Reply to this email directly or view it on GitHub
#256 (comment).

@sharmaeklavya2
Contributor

I searched for 'python3' and 'python 3' in Zulip's commit history. I still can't understand what you mean by '1 commit per fixer'. Is this what you mean by fixer: https://docs.python.org/3/library/2to3.html#fixers?

@sharmaeklavya2
Contributor

Thanks. I got it now.

@sharmaeklavya2
Contributor

./tools/travis/py3k only showed these diffs:

  • zulip_tools.py in lib2to3.fixes.fix_except
  • zerver/lib/cache.py in libfuturize.fixes.fix_print_with_import

Manually doing a git diff however showed many more changes:

    modified:   analytics/management/commands/analyze_mit.py
    modified:   api/zulip/__init__.py
    modified:   bots/zephyr_mirror_backend.py
    modified:   puppet/zulip_internal/files/postgresql/pg_backup_and_purge.py
    modified:   zerver/lib/actions.py
    modified:   zerver/lib/alert_words.py
    modified:   zerver/lib/bugdown/__init__.py
    modified:   zerver/lib/event_queue.py
    modified:   zerver/lib/rest.py
    modified:   zerver/lib/statistics.py
    modified:   zerver/management/commands/realm_emoji.py
    modified:   zerver/test_bugdown.py
    modified:   zerver/test_signup.py
    modified:   zerver/test_subs.py
    modified:   zerver/views/messages.py
    modified:   zerver/views/streams.py
    modified:   zilencer/error_notify.py
    modified:   zproject/settings.py

It seems that the script is not working like it is supposed to.

@sharmaeklavya2
Contributor

I ran the above py3k test using python3 venv. When I ran it using python2 venv, the output of the script is the same but git diff didn't show any change.

@timabbott
Member
@sharmaeklavya2
Contributor

I also noticed that when running py3k using a python2 venv, the last few tests are not run, which are:

libfuturize.fixes.fix_division_safe
libmodernize.fixes.fix_file
libmodernize.fixes.fix_filter
libmodernize.fixes.fix_imports_six
libmodernize.fixes.fix_input_six
libmodernize.fixes.fix_int_long_tuple
libmodernize.fixes.fix_map
libmodernize.fixes.fix_raise_six
libmodernize.fixes.fix_xrange_six
libmodernize.fixes.fix_zip
libmodernize.fixes.fix_unicode_type
libpasteurize.fixes.fix_newstyle
libmodernize.fixes.fix_dict_six

Also after running py3k on a python3 venv, on observing the output of git diff (which I run manually), it seems that all changes come from libmodernize.fixes.fix_dict_six.

@sharmaeklavya2
Contributor

When invoked on a directory, futurize only works on .py files in that directory. If it is invoked directly on a file, the file's extension doesn't matter.

I'll check the mime-type of all files and call futurize only on those files which have the mime-type text/x-python. Mime type can be found using the command file -b --mime-type NAME_OF_FILE.

@sharmaeklavya2
Contributor

I have written a script using which uses ** in bash to recursively operate on files. However, it takes around 4 minutes for a fixer to run on all python files. With 40 fixers, this can take around 3 hours.

@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 10, 2016
@sharmaeklavya2 sharmaeklavya2 Fixed tools/travis/py3k
Previously tools/travis/py3k only ran on files which end with '.py'.
Now it runs on all python files, even those without a file extension.

Previously futurize was being run on a directory. futurize only scans .py files
if used on a directory. But there are python files which don't have the .py
extension. To deal with this, we need to call futurize on files individually.
Now futurize is called on files which .py extension or which are executable files
with the 'text/x-python' mime-type. The py3k script can also be used to apply a
fixer to the project. The fixer should be sent as the first command-line argument.

Refer to #256
8f8847b
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 10, 2016
@sharmaeklavya2 sharmaeklavya2 Apply Python 3 futurize transform lib2to3.fixes.fix_has_key
Refer to #256
18131ec
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 10, 2016
@sharmaeklavya2 sharmaeklavya2 Apply Python 3 futurize transform lib2to3.fixes.fix_idioms
Refer to #256
8f90522
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 10, 2016
@sharmaeklavya2 sharmaeklavya2 Apply Python 3 futurize transform libfuturize.fixes.fix_print_with_im…
…port

Refer #256
55739c9
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 10, 2016
@sharmaeklavya2 sharmaeklavya2 Apply Python 3 futurize transform libfuturize.fixes.fix_raise
Refer to #256
4fe0636
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 10, 2016
@sharmaeklavya2 sharmaeklavya2 Apply Python 3 futurize transform libmodernize.fixes.fix_basestring
Refer to #256
f7a4fd6
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 10, 2016
@sharmaeklavya2 sharmaeklavya2 Apply Python 3 futurize transform libfuturize.fixes.fix_division_safe
Refer to #256
1a6382e
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 10, 2016
@sharmaeklavya2 sharmaeklavya2 Apply Python 3 futurize transform libmodernize.fixes.fix_file
Refer to #256
c40596c
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 10, 2016
@sharmaeklavya2 sharmaeklavya2 Apply Python 3 futurize transform libmodernize.fixes.fix_filter
Refer to #256
c12d3c4
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 10, 2016
@sharmaeklavya2 sharmaeklavya2 Apply Python 3 futurize transform libmodernize.fixes.fix_imports_six
Refer to #256
b84dff6
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 10, 2016
@sharmaeklavya2 sharmaeklavya2 Apply Python 3 futurize transform libmodernize.fixes.fix_map
Refer to #256
126eeb9
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 10, 2016
@sharmaeklavya2 sharmaeklavya2 Apply Python 3 futurize transform libmodernize.fixes.fix_xrange_six
Refer to #256
3fd796e
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 10, 2016
@sharmaeklavya2 sharmaeklavya2 Apply Python 3 futurize transform libmodernize.fixes.fix_unicode_type
Refer to #256
33061a9
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 10, 2016
@sharmaeklavya2 sharmaeklavya2 Apply Python 3 futurize transform libpasteurize.fixes.fix_newstyle
Refer to #256
2b625f1
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 10, 2016
@sharmaeklavya2 sharmaeklavya2 Apply Python 3 futurize transform libmodernize.fixes.fix_dict_six
Refer to #256
21a1146
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 10, 2016
@sharmaeklavya2 sharmaeklavya2 Change shebangs from python2.7 to python
Refer to #256
1da1a68
@sharmaeklavya2
Contributor

All futurize transforms done!

@sharmaeklavya2
Contributor

I tried to run tests now. tools/lint-all has some unicode encode-decode related errors. tools/test-migrations and tools/test-backend failed as soon as they started because ImportError: No module named 'ldap'. I haven't yet installed python-ldap or pyldap.

@timabbott timabbott added a commit to timabbott/zulip that referenced this issue Mar 11, 2016
@sharmaeklavya2 @timabbott sharmaeklavya2 + timabbott Apply Python 3 futurize transform lib2to3.fixes.fix_has_key
Refer to #256
ab72874
@timabbott timabbott added a commit to timabbott/zulip that referenced this issue Mar 11, 2016
@sharmaeklavya2 @timabbott sharmaeklavya2 + timabbott Apply Python 3 futurize transform lib2to3.fixes.fix_idioms
Refer to #256
4fb549a
@timabbott timabbott added a commit to timabbott/zulip that referenced this issue Mar 11, 2016
@sharmaeklavya2 @timabbott sharmaeklavya2 + timabbott Apply Python 3 futurize transform libfuturize.fixes.fix_raise
Refer to #256
1941201
@timabbott timabbott added a commit to timabbott/zulip that referenced this issue Mar 11, 2016
@sharmaeklavya2 @timabbott sharmaeklavya2 + timabbott Apply Python 3 futurize transform libmodernize.fixes.fix_file
Refer to #256
d3b63f9
@timabbott timabbott added a commit to timabbott/zulip that referenced this issue Mar 11, 2016
@sharmaeklavya2 @timabbott sharmaeklavya2 + timabbott Apply Python 3 futurize transform libmodernize.fixes.fix_map
Refer to #256
aa505b0
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 11, 2016
@sharmaeklavya2 sharmaeklavya2 Partially apply Python 3 transform libpasteurize.fixes.fix_newstyle
Refer to #256
f40ca03
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 11, 2016
@sharmaeklavya2 sharmaeklavya2 Made divisions python 3 compatible
Refer to #256
a68dc6c
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 11, 2016
@sharmaeklavya2 sharmaeklavya2 Partially apply Python 3 libmodernize.fixes.fix_dict_six
Refer to #256
ce81998
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 12, 2016
@sharmaeklavya2 sharmaeklavya2 Partially apply Python 3 transform libpasteurize.fixes.fix_newstyle
Refer to #256
3784e65
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 12, 2016
@sharmaeklavya2 sharmaeklavya2 Made divisions python 3 compatible
Refer to #256
87ba0a3
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 12, 2016
@sharmaeklavya2 sharmaeklavya2 Partially apply Python 3 libmodernize.fixes.fix_dict_six
Refer to #256
7fb7bfa
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 12, 2016
@sharmaeklavya2 sharmaeklavya2 Partially apply Python 3 transform libpasteurize.fixes.fix_newstyle
Refer to #256
14130a8
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 12, 2016
@sharmaeklavya2 sharmaeklavya2 Made divisions python 3 compatible
Refer to #256
502ff43
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 12, 2016
@sharmaeklavya2 sharmaeklavya2 Partially apply Python 3 libmodernize.fixes.fix_dict_six
Refer to #256
a20a845
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 12, 2016
@sharmaeklavya2 sharmaeklavya2 Partially apply Python 3 libmodernize.fixes.fix_dict_six
Refer to #256
0e2fb85
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 12, 2016
@sharmaeklavya2 sharmaeklavya2 Fix tools/travis/py3k
Previously tools/travis/py3k only ran on '.py' files.
Now it also runs on executable python scripts.

Previously futurize was being run on a directory.
futurize only scans .py files if used on a directory.
But there are python files which don't have the .py
extension. To deal with this, we need to call
futurize on files individually. Now futurize is
called on files with .py extension or which are
executable files with the 'text/x-python' mime-type.
The py3k script can also be used to apply a fixer
to the project. The fixer should be sent as the
first command-line argument.

Refer to #256
547a120
@sharmaeklavya2
Contributor

I tried to get a python3 venv working. Most packages had a python 3 port available. tools/test-backend runs well on the first 17 tests and gave this error on the following test

Running Bugdown test camo
Traceback (most recent call last):
  File "/mnt/Code/git_repos/forked/zulip/zerver/lib/test_runner.py", line 77, in run_test
    test_method()
  File "/mnt/Code/git_repos/forked/zulip/zerver/test_bugdown.py", line 156, in test_bugdown_fixtures
    self.assertEqual(converted, test['expected_output'])
  File "/usr/lib/python3.4/unittest/case.py", line 800, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python3.4/unittest/case.py", line 1173, in assertMultiLineEqual
    self.fail(self._formatMessage(msg, standardMsg))
  File "/usr/lib/python3.4/unittest/case.py", line 645, in fail
    raise self.failureException(msg)
AssertionError: '<p>G[438 chars]de79/b\'687474703a2f2f7777772e676f6f676c652e63[60 chars]div>' != '<p>G[438 chars]de79/687474703a2f2f7777772e676f6f676c652e636f6[55 chars]div>'
  <p>Google logo today: <a href="http://www.google.com/images/srpr/logo4w.png" target="_blank" title="http://www.google.com/images/srpr/logo4w.png">http://www.google.com/images/srpr/logo4w.png</a></p>
- <div class="message_inline_image"><a href="http://www.google.com/images/srpr/logo4w.png" target="_blank" title="http://www.google.com/images/srpr/logo4w.png"><img src="https://external-content.zulipcdn.net/7b6552b60c635e41e8f6daeb36d88afc4eabde79/b'687474703a2f2f7777772e676f6f676c652e636f6d2f696d616765732f737270722f6c6f676f34772e706e67'"></a></div>?                                                                                                                                                                                                                                                        --                                                                                        -
+ <div class="message_inline_image"><a href="http://www.google.com/images/srpr/logo4w.png" target="_blank" title="http://www.google.com/images/srpr/logo4w.png"><img src="https://external-content.zulipcdn.net/7b6552b60c635e41e8f6daeb36d88afc4eabde79/687474703a2f2f7777772e676f6f676c652e636f6d2f696d616765732f737270722f6c6f676f34772e706e67"></a></div>
FAILED!

I traced this down to line 144 in bugdown-data.json. What is bugdown and what is camo?

@timabbott
Member

Bugdown is the Zulip markdown parser (implemented as an extension to python-markdown), zerver/lib/bugdown/__init__.py is the main implementation file, and bugdown-data.json is the test data set for it. Camo is a https image proxy that Zulip uses to avoid mixed-content warnings.

Looking at that error, it appears it's an unicode bug of some sort -- b'' is getting included in part fo the URL when generating that URL. Search the Bugdown code for message_inline_image and you'll find the implementation that is likely responsible for this python 3 compatibility bug.

@sharmaeklavya2
Contributor

I fixed it. InlineHttpsProcessor.run had this line:

img.set("src", "%s%s/%s" % (settings.CAMO_URI, digest, hex_encoded_url))

I changed it to:

img.set("src", "%s%s/%s" % (settings.CAMO_URI, digest, hex_encoded_url.decode("utf-8")))
@sharmaeklavya2
Contributor

Now I'm facing this error:

Running zerver.test_email_mirror.TestMissedHuddleMessageEmailMessages.test_receive_missed_huddle_message_email_messages
2016-03-13 05:25:30,008 ERROR    Internal Server Error: /accounts/login/
Traceback (most recent call last):
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/core/handlers/base.py", line 164, in get_response
    response = response.render()
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/response.py", line 158, in render
    self.content = self.rendered_content

... (I shortened the traceback because it was too long)

  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 1331, in import_library
    mod = import_module(taglib_module)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1467, in exec_module
  File "<frozen importlib._bootstrap>", line 1572, in get_code
  File "<frozen importlib._bootstrap>", line 1532, in source_to_code
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/pipeline/templatetags/compressed.py", line 102
    raise template.TemplateSyntaxError, '%r requires exactly one argument: the name of a group in the PIPELINE_CSS setting' % token.split_contents()[0]

SyntaxError: invalid syntax

The problem is arising because django-pipeline==1.2.2 is not python 3 compatible. The latest version 1.6.7 is backwards incompatible with 1.2.2.

@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 13, 2016
@sharmaeklavya2 sharmaeklavya2 Handle unicode properly in camo
In Bugdown's InlineHttpsProcessor.run,
hex_encoded_url was not being decoded to
utf-8. This was causing problems in python 3.

Refer to #256
9665abd
@sharmaeklavya2
Contributor

I noticed that in python 3 tests in zerver/tests.py are not being run. Any idea why this could be happening?

@sharmaeklavya2
Contributor
sharmaeklavya2 commented Mar 13, 2016 edited

I have made a small report about python 3 support of controversial packages. I hope this will help. Here 'pypi python support' means the version of python supported by the package according to lines of the form Programming Language :: Python :: [\d.]+ on the package's pypi page.

  • fonttools
    • pypi python support: -
    • Supports py3.3+ according to README
    • tools/setup/emoji_dump/emoji_dump.py is the only place where fonttools is being used. I checked that that script runs fine on python 3.
  • py3dns
    • version: latest is 3.1.0
    • pypi python support: 3
    • probably compatible with pydns because dns-related errors on running tools/test-backend disappeared after installing py3dns
  • django-auth-ldap - Fix confirmed in #1175.
  • django-pipeline - Fixed in #1176.
  • python-twitter - Fixed in #1308. Confirmed using both manual and automated testing.
  • apns-client - superseded by PyAPNS in #980. Needs manual testing.
  • Twisted - Not yet ported to python 3.
  • gcm-client
    • version: We use 0.1.4 which is latest
    • pypi python support: -
    • my version might support py3.4. It is taken from a fork in hg which I found here.
  • mandrill
    • version: We use 1.0.57 which is latest
    • pypi python support: -
    • Supports python 2.6-2.7, 3.0-3.4 according to a pull-request
@sharmaeklavya2
Contributor

I deleted all test files except tests.py and ran tools/test-backend. It then worked and showed the same django-pipeline error as above on the very first test.

It seems we have no option but to port our code to the new django-pipeline before any more progress can be had on python 3 support.

@timabbott
Member

Great, thanks for putting together that analysis @sharmaeklavya2 !

I think there are two angles for how to make forward progress on python 3 support from here:

  • You can run the tests with test-backend --nonfatal-errors to see the whole set of which tests pass and fail, and thus fix those issues that are unrelated to the django-pipeline version issue.
  • I agree that we'll need to do the django-pipeline upgrade, and that's probably the most disruptive of the things we will need to upgrade. If I recall correctly, there are several API changes in the newer versions of django-pipeline, so it may require some effort to complete; I'd probably do that upgrade on a separate branch from the rest of python 3 support since it's an upgrade with value independent from the other python 3 changes and might be ready to merge at a different time from the rest.

I don't know why the zerver/tests.py tests would not be running -- are you sure what you're seeing isn't the tests running in a different order (and failing before the zerver/tests.py tests are reached)?

@sharmaeklavya2
Contributor

Well yeah you were right about tests.py not running because it's due order is different. I did a small experiment to find that out. Nevertheless, it is failing with the same error message (django-pipeline).

@timabbott
Member
@sharmaeklavya2
Contributor
Running zerver.tests.ActivateTest.test_api
2016-03-13 21:34:37,350 ERROR    Internal Server Error: /accounts/login/
Traceback (most recent call last):
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/core/handlers/base.py", line 164, in get_response
    response = response.render()
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/response.py", line 158, in render
    self.content = self.rendered_content
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/response.py", line 135, in rendered_content
    content = template.render(context, self._request)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/backends/django.py", line 74, in render
    return self.template.render(context)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 209, in render
    return self._render(context)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/test/utils.py", line 96, in instrumented_test_render
    return self.nodelist.render(context)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 903, in render
    bit = self.render_node(node, context)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/debug.py", line 79, in render_node
    return node.render(context)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/loader_tags.py", line 113, in render
    compiled_parent = self.get_parent(context)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/loader_tags.py", line 110, in get_parent
    return context.template.engine.get_template(parent)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/engine.py", line 167, in get_template
    template, origin = self.find_template(template_name, dirs)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/engine.py", line 141, in find_template
    source, display_name = loader(name, dirs)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/loaders/base.py", line 13, in __call__
    return self.load_template(template_name, template_dirs)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/loaders/base.py", line 23, in load_template
    template = Template(source, origin, template_name, self.engine)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 190, in __init__
    self.nodelist = engine.compile_string(template_string, origin)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/engine.py", line 261, in compile_string
    return parser.parse()
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 341, in parse
    compiled_result = compile_func(self, token)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/loader_tags.py", line 210, in do_extends
    nodelist = parser.parse()
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 341, in parse
    compiled_result = compile_func(self, token)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/defaulttags.py", line 1155, in load
    lib = get_library(taglib)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 1392, in get_library
    lib = import_library(taglib_module)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 1331, in import_library
    mod = import_module(taglib_module)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1467, in exec_module
  File "<frozen importlib._bootstrap>", line 1572, in get_code
  File "<frozen importlib._bootstrap>", line 1532, in source_to_code
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/pipeline/templatetags/compressed.py", line 102
    raise template.TemplateSyntaxError, '%r requires exactly one argument: the name of a group in the PIPELINE_CSS setting' % token.split_contents()[0]
                                      ^
SyntaxError: invalid syntax
Traceback (most recent call last):
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/core/handlers/base.py", line 164, in get_response
    response = response.render()
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/response.py", line 158, in render
    self.content = self.rendered_content
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/response.py", line 135, in rendered_content
    content = template.render(context, self._request)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/backends/django.py", line 74, in render
    return self.template.render(context)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 209, in render
    return self._render(context)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/test/utils.py", line 96, in instrumented_test_render
    return self.nodelist.render(context)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 903, in render
    bit = self.render_node(node, context)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/debug.py", line 79, in render_node
    return node.render(context)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/loader_tags.py", line 113, in render
    compiled_parent = self.get_parent(context)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/loader_tags.py", line 110, in get_parent
    return context.template.engine.get_template(parent)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/engine.py", line 167, in get_template
    template, origin = self.find_template(template_name, dirs)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/engine.py", line 141, in find_template
    source, display_name = loader(name, dirs)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/loaders/base.py", line 13, in __call__
    return self.load_template(template_name, template_dirs)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/loaders/base.py", line 23, in load_template
    template = Template(source, origin, template_name, self.engine)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 190, in __init__
    self.nodelist = engine.compile_string(template_string, origin)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/engine.py", line 261, in compile_string
    return parser.parse()
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 341, in parse
    compiled_result = compile_func(self, token)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/loader_tags.py", line 210, in do_extends
    nodelist = parser.parse()
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 341, in parse
    compiled_result = compile_func(self, token)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/defaulttags.py", line 1155, in load
    lib = get_library(taglib)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 1392, in get_library
    lib = import_library(taglib_module)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 1331, in import_library
    mod = import_module(taglib_module)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1467, in exec_module
  File "<frozen importlib._bootstrap>", line 1572, in get_code
  File "<frozen importlib._bootstrap>", line 1532, in source_to_code
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/pipeline/templatetags/compressed.py", line 102
    raise template.TemplateSyntaxError, '%r requires exactly one argument: the name of a group in the PIPELINE_CSS setting' % token.split_contents()[0]
                                      ^
SyntaxError: invalid syntax

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/Code/git_repos/forked/zulip/zerver/lib/test_runner.py", line 77, in run_test
    test_method()
  File "/mnt/Code/git_repos/forked/zulip/zerver/tests.py", line 347, in test_api
    self.login('othello@zulip.com')
  File "/mnt/Code/git_repos/forked/zulip/zerver/lib/test_helpers.py", line 216, in login
    {'username':email, 'password':password})
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/test/client.py", line 512, in post
    secure=secure, **extra)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/test/client.py", line 313, in post
    secure=secure, **extra)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/test/client.py", line 379, in generic
    return self.request(**r)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/test/client.py", line 448, in request
    response = self.handler(environ)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/test/client.py", line 122, in __call__
    response = self.get_response(request)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/core/handlers/base.py", line 218, in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/core/handlers/base.py", line 268, in handle_uncaught_exception
    return callback(request, **param_dict)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/utils/decorators.py", line 110, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/views/defaults.py", line 45, in server_error
    return http.HttpResponseServerError(template.render())
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/backends/django.py", line 74, in render
    return self.template.render(context)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 209, in render
    return self._render(context)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/test/utils.py", line 96, in instrumented_test_render
    return self.nodelist.render(context)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 903, in render
    bit = self.render_node(node, context)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/debug.py", line 79, in render_node
    return node.render(context)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/loader_tags.py", line 113, in render
    compiled_parent = self.get_parent(context)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/loader_tags.py", line 110, in get_parent
    return context.template.engine.get_template(parent)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/engine.py", line 167, in get_template
    template, origin = self.find_template(template_name, dirs)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/engine.py", line 141, in find_template
    source, display_name = loader(name, dirs)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/loaders/base.py", line 13, in __call__
    return self.load_template(template_name, template_dirs)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/loaders/base.py", line 23, in load_template
    template = Template(source, origin, template_name, self.engine)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 190, in __init__
    self.nodelist = engine.compile_string(template_string, origin)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/engine.py", line 261, in compile_string
    return parser.parse()
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 341, in parse
    compiled_result = compile_func(self, token)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/loader_tags.py", line 210, in do_extends
    nodelist = parser.parse()
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 341, in parse
    compiled_result = compile_func(self, token)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/defaulttags.py", line 1155, in load
    lib = get_library(taglib)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 1392, in get_library
    lib = import_library(taglib_module)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/django/template/base.py", line 1331, in import_library
    mod = import_module(taglib_module)
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1467, in exec_module
  File "<frozen importlib._bootstrap>", line 1572, in get_code
  File "<frozen importlib._bootstrap>", line 1532, in source_to_code
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/mnt/Code/git_repos/forked/zvenv3/lib/python3.4/site-packages/pipeline/templatetags/compressed.py", line 102
    raise template.TemplateSyntaxError, '%r requires exactly one argument: the name of a group in the PIPELINE_CSS setting' % token.split_contents()[0]
                                      ^
SyntaxError: invalid syntax
FAILED!
@sharmaeklavya2
Contributor

Thanks for the --nonfatal-errors switch. I found many errors not related to django-pipeline. They seem easy to fix. django-pipeline is the only dependency causing problems as of now.

@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 14, 2016
@sharmaeklavya2 sharmaeklavya2 Partially apply Python 3 libmodernize.fixes.fix_dict_six
Refer to #256
c4aaf2c
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 14, 2016
@sharmaeklavya2 sharmaeklavya2 Fix tools/travis/py3k
Previously tools/travis/py3k only ran on '.py' files.
Now it also runs on executable python scripts.

Previously futurize was being run on a directory.
futurize only scans .py files if used on a directory.
But there are python files which don't have the .py
extension. To deal with this, we need to call
futurize on files individually. Now futurize is
called on files with .py extension or which are
executable files with the 'text/x-python' mime-type.
The py3k script can also be used to apply a fixer
to the project. The fixer should be sent as the
first command-line argument.

Refer to #256
2057224
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 14, 2016
@sharmaeklavya2 sharmaeklavya2 Handle unicode properly in camo
In Bugdown's InlineHttpsProcessor.run,
hex_encoded_url was not being decoded to
utf-8. This was causing problems in python 3.

Refer to #256
8983416
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 14, 2016
@sharmaeklavya2 sharmaeklavya2 Fix python 3 decode error in zerver/tests.py
Refer to #256
35119b1
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 14, 2016
@sharmaeklavya2 sharmaeklavya2 Removed calls to ifilterfalse
Replaced calls to ifilterfalse by list comprehensions
because ifilterfalse is not part of python 3.
Also changed some lists to sets for faster lookup.

Refer to #256
19d3bf2
@sharmaeklavya2
Contributor

In zerver/models.py, __repr__ and __str__ methods of models return bytes instead of str. This is okay in python 2 but it's wrong in python 3 and gives a TypeError.

For eg, we have this in the Stream model:

    def __repr__(self):
        return (u"<Stream: %s>" % (self.name,)).encode("utf-8")

I think we can safely change it to

    def __repr__(self):
        return "<Stream: %s>" % (self.name,)

I changed it to that and tools/test-backend is not giving any error on python 2.

But I thought maybe there is a reason why it is written like this (because why would the developers who wrote this unnecessarily complicate things?). Is there a reason?

@timabbott
Member

Yeah, I think there was a reason -- the reproducer for the issue that resolved is this:

from zerver.models import *; print(Stream(name=u"☃", realm=get_realm("zulip.com")))

@timabbott
Member

Re the django-pipeline issue, that traceback makes sense; I think we basically will have to resolve that by upgrading django-pipeline.

@sharmaeklavya2
Contributor

Django's way of handling __str__ and __unicode__ is the cleanest I have found so far. I think this is what we should do for our models.

@timabbott
Member
@sharmaeklavya2
Contributor

I noticed that we are using an old version of pika (0.9.14) which does not support Python 3. The latest version (0.10.0) has Python 3 support. I have updated the packages report above.

@sharmaeklavya2
Contributor

I have ported Zulip to django-pipeline 1.6.7 (which is the latest version) from 1.2.2. See pull request #545.

@sharmaeklavya2
Contributor

I have been using the latest version of pika for a few days. No problems have come up so far, except that it sometimes throws weird errors when I stop tools/run-dev.py with Ctrl + C.

@timabbott
Member

Cool! Might need to track down those errors, though they might go away with #367...

@timabbott timabbott added a commit that referenced this issue Mar 19, 2016
@sharmaeklavya2 @timabbott sharmaeklavya2 + timabbott Handle unicode properly in camo.
In Bugdown's InlineHttpsProcessor.run, hex_encoded_url was not being
decoded to utf-8. This was causing problems in python 3.

Refer to #256.
186efc6
@timabbott timabbott added a commit that referenced this issue Mar 19, 2016
@sharmaeklavya2 @timabbott sharmaeklavya2 + timabbott Removed calls to ifilterfalse.
Replaced calls to ifilterfalse by list comprehensions because
ifilterfalse is not part of python 3.  Also changed some lists to sets
for faster lookup.

Refer to #256.
176c507
@ashishk1994 ashishk1994 added a commit to ashishk1994/zulip that referenced this issue Mar 20, 2016
@sharmaeklavya2 @ashishk1994 sharmaeklavya2 + ashishk1994 Handle unicode properly in camo.
In Bugdown's InlineHttpsProcessor.run, hex_encoded_url was not being
decoded to utf-8. This was causing problems in python 3.

Refer to #256.
33d2eed
@ashishk1994 ashishk1994 added a commit to ashishk1994/zulip that referenced this issue Mar 20, 2016
@sharmaeklavya2 @ashishk1994 sharmaeklavya2 + ashishk1994 Fix python 3 decode error in zerver/tests.py.
Refer to #256.
9b5374b
@ashishk1994 ashishk1994 added a commit to ashishk1994/zulip that referenced this issue Mar 20, 2016
@sharmaeklavya2 @ashishk1994 sharmaeklavya2 + ashishk1994 Removed calls to ifilterfalse.
Replaced calls to ifilterfalse by list comprehensions because
ifilterfalse is not part of python 3.  Also changed some lists to sets
for faster lookup.

Refer to #256.
4b815a4
@ashishk1994 ashishk1994 added a commit to ashishk1994/zulip that referenced this issue Mar 20, 2016
@sharmaeklavya2 @ashishk1994 sharmaeklavya2 + ashishk1994 Partially apply Python 3 libmodernize.fixes.fix_dict_six.
Refer to #256
998e03f
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 20, 2016
@sharmaeklavya2 sharmaeklavya2 Change shebangs from python2.7 to python.
Refer to #256.
6f5a664
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 20, 2016
@sharmaeklavya2 sharmaeklavya2 Replace python2.7 with python in comments and subprocess calls.
Refer to #256.
2efc2c7
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 20, 2016
@sharmaeklavya2 sharmaeklavya2 Make py3k run fixers on executable scripts too.
Also display the time whenever futurize or python-modernize is
invoked so that we can measure performance.

Refer to #256.
7dd8462
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 20, 2016
@sharmaeklavya2 sharmaeklavya2 Make py3k run fixers on executable scripts too.
Also display the time whenever futurize or python-modernize is
invoked so that we can measure performance.

Refer to #256.
101bfcf
@sharmaeklavya2 sharmaeklavya2 added a commit to sharmaeklavya2/zulip that referenced this issue Mar 20, 2016
@sharmaeklavya2 sharmaeklavya2 Make py3k run fixers on executable scripts too.
Also display the time whenever futurize or python-modernize is
invoked so that we can measure performance.

Refer to #256.
5b0cfd6
@TomaszKolek TomaszKolek pushed a commit to TomaszKolek/zulip that referenced this issue Mar 20, 2016
@sharmaeklavya2 sharmaeklavya2 + Tomasz Kolek Handle unicode properly in camo.
In Bugdown's InlineHttpsProcessor.run, hex_encoded_url was not being
decoded to utf-8. This was causing problems in python 3.

Refer to #256.
2198f74
@TomaszKolek TomaszKolek pushed a commit to TomaszKolek/zulip that referenced this issue Mar 20, 2016
@sharmaeklavya2 sharmaeklavya2 + Tomasz Kolek Fix python 3 decode error in zerver/tests.py.
Refer to #256.
e57046a
@TomaszKolek TomaszKolek pushed a commit to TomaszKolek/zulip that referenced this issue Mar 20, 2016
@sharmaeklavya2 sharmaeklavya2 + Tomasz Kolek Removed calls to ifilterfalse.
Replaced calls to ifilterfalse by list comprehensions because
ifilterfalse is not part of python 3.  Also changed some lists to sets
for faster lookup.

Refer to #256.
aacec9a
@TomaszKolek TomaszKolek pushed a commit to TomaszKolek/zulip that referenced this issue Mar 20, 2016
@sharmaeklavya2 sharmaeklavya2 + Tomasz Kolek Partially apply Python 3 libmodernize.fixes.fix_dict_six.
Refer to #256
4d4d47f
@TomaszKolek TomaszKolek pushed a commit to TomaszKolek/zulip that referenced this issue Mar 21, 2016
@sharmaeklavya2 sharmaeklavya2 + Tomasz Kolek Handle unicode properly in camo.
In Bugdown's InlineHttpsProcessor.run, hex_encoded_url was not being
decoded to utf-8. This was causing problems in python 3.

Refer to #256.
cfddb59
@TomaszKolek TomaszKolek pushed a commit to TomaszKolek/zulip that referenced this issue Mar 21, 2016
@sharmaeklavya2 sharmaeklavya2 + Tomasz Kolek Fix python 3 decode error in zerver/tests.py.
Refer to #256.
8d45c52
@TomaszKolek TomaszKolek pushed a commit to TomaszKolek/zulip that referenced this issue Mar 21, 2016
@sharmaeklavya2 sharmaeklavya2 + Tomasz Kolek Removed calls to ifilterfalse.
Replaced calls to ifilterfalse by list comprehensions because
ifilterfalse is not part of python 3.  Also changed some lists to sets
for faster lookup.

Refer to #256.
d465b2a
@TomaszKolek TomaszKolek pushed a commit to TomaszKolek/zulip that referenced this issue Mar 21, 2016
@sharmaeklavya2 sharmaeklavya2 + Tomasz Kolek Partially apply Python 3 libmodernize.fixes.fix_dict_six.
Refer to #256
3cdf7d0
@sharmaeklavya2
Contributor

I saw that the file tools/python-proxy was giving a UnicodeDecodeError with python3. When I opened the file, I found that it has encoding cp1252. There are a few special characters (Spanish?) in that file where python 3 throws up. Python2 does not give that error, but it is not able to display those special characters correctly on my terminal.

There might be a few more files like this. Maybe we should convert their encodings to utf-8.

@timabbott
Member
@sharmaeklavya2
Contributor

After grepping for the -*- coding: something -*- headers, I think python-proxy is the only non-utf-8 file that we have.

@timabbott
Member
@sharmaeklavya2
Contributor

I have changed tools/python-proxy's encoding to utf-8.

I can't see where this file is being used. I grepped for 'python-proxy' and the only result that came up was THIRDPARTY.

If we're not using this file and we're likely to not use it in the future, then I think it is better to just delete it.

@timabbott
Member
@sharmaeklavya2
Contributor

Ok. I'll delete it and send a pull request.

@disissid disissid pushed a commit to disissid/zulip that referenced this issue Apr 3, 2016
@sharmaeklavya2 sharmaeklavya2 + Kumar Handle unicode properly in camo.
In Bugdown's InlineHttpsProcessor.run, hex_encoded_url was not being
decoded to utf-8. This was causing problems in python 3.

Refer to #256.
648ac84
@disissid disissid pushed a commit to disissid/zulip that referenced this issue Apr 3, 2016
@sharmaeklavya2 sharmaeklavya2 + Kumar Fix python 3 decode error in zerver/tests.py.
Refer to #256.
3c0f1d8
@disissid disissid pushed a commit to disissid/zulip that referenced this issue Apr 3, 2016
@sharmaeklavya2 sharmaeklavya2 + Kumar Removed calls to ifilterfalse.
Replaced calls to ifilterfalse by list comprehensions because
ifilterfalse is not part of python 3.  Also changed some lists to sets
for faster lookup.

Refer to #256.
cc1471c
@disissid disissid pushed a commit to disissid/zulip that referenced this issue Apr 3, 2016
@sharmaeklavya2 sharmaeklavya2 + Kumar Partially apply Python 3 libmodernize.fixes.fix_dict_six.
Refer to #256
d72181c
@timabbott timabbott added this to the 2016 roadmap milestone Apr 29, 2016
@timabbott
Member

@sharmaeklavya2 I've been thinking a lot about how to push forward the Python 3 project. One of the big things we're going to need to do for that project is upgrade a lot of the Zulip dependencies to newer versions that support Python 3. Currently, projects to do this get stuck, because we package the Zulip dependencies differently in production vs. in development. In development, we use requirements.txt, and have a model that makes upgrades work cleanly (run provision.py and it will automatically create a new venv named after the hash of requirements.txt). In production, we have an apt repository that is shared by every version of Zulip ever published, which means that it's very difficult to do updates to Python modules that are backwards-incompatible without breaking old versions of the software.

I think the right solution here is to migrate Zulip production to use the virtualenv model we have in development (which is #717).

What do you think?

@sharmaeklavya2
Contributor

Yeah I think virtualenv model would be better for production.

@sharmaeklavya2
Contributor
sharmaeklavya2 commented Jul 11, 2016 edited

I realized that some modules in Twisted are missing in python 3. On looking it up I found this - http://twisted.readthedocs.io/en/twisted-16.1.1/core/howto/python3.html.

According to this document, only python 3 compatible sub-packages of Twisted are installed in python 3.

We are using the following modules in Zulip. I have ticked the modules which are available on python 3 in the latest version of Twisted - 16.3.0:

  • twisted.internet.reactor
  • twisted.internet.protocol
  • twisted.internet.ssl
  • twisted.web.proxy
  • twisted.web.server
  • twisted.web.resource
  • twisted.web.http
  • twisted.mail.imap4

The python 3 compatibility status of Twisted 15.2.1 (our current version), is the same for the above sub-packages, except that twisted.web.proxy is not available.

twisted.mail.imap4 is used in zerver/management/commands/email-mirror.py.

twisted.web.server and twisted.web.http are listed in the documentation to not be completely compatible, but they are included in Twisted and might work. They are used in tools/run-dev.py.

@timabbott
Member
timabbott commented Jul 11, 2016 edited

OK we'll need to upgrade Twisted for run-dev.py since that uses twisted.web.proxy as well.

I would guess that if they're partially compatible, it'd be fine, since run-dev.py isn't a very intensive use of Twisted.

@timabbott
Member

For twisted.mail.imap4: there's basically two options here:

  • We could try to replace twisted.mail.imap4, since we only use it for one thing (the IMAP mode of the email mirror) and it's possible https://docs.python.org/3/library/imaplib.html would cover that use case reasonably well. This would remove our need for twisted in production altogether, which is probably net nice.
  • We could try to fix twisted.mail.imap4 to work with Python 3. I'd worry a bit that could be a large project:
$ wc /srv/zulip-venv/lib/python2.7/site-packages/twisted/mail/imap4.py
  6253  22267 214306 /srv/zulip-venv/lib/python2.7/site-packages/twisted/mail/imap4.py

but it's possible the work has already been partially done by the Twisted community. I'd consider dropping by the Twisted IRC channel (see https://twistedmatrix.com/trac/wiki/TwistedCommunity) and asking how close that is to being done.

  • We could skip the email mirror tests when run with Python 3 for now and leave that component as Python 2 only. Not excited about that in the long term but could be reasonable for trying to get everything else working happily.
@timabbott
Member

This issue is basically done -- all of our test suites but the end-to-end production installation one pass with Python 3. The remaining list is:

  • run-dev.py doesn't support Python 3
  • build-release-tarball (#1752)
  • Testing actually running Zulip in Python 3 in a production environment. I'd start with getting the production test suite in Travis CI passing with Python 3.
@timabbott
Member

The only thing now left here is to actually start running Zulip in a production server on Python 3.

I think in order to do that, the next step is to add a way to configure Zulip to actually run using Python 3. I think what actually controls what we use in production is:

  • the code in scripts/lib/setup_path_on_import.py, since that's used by manage.py and all of our other tools to setup the production virtualenv, and that currently hardcodes the path to a Python 2 virtualenv.
  • the code in create-production-venv and its caller, which use #/usr/bin/env python to figure out which Python version to actually use when creating virtualenvs
  • the code in puppet/zulip/files/supervisor/conf.d/zulip.conf, which hardcodes using python explicitly, and thus will run whatever python is first in PATH from whatever environment supervisord was started in. This can be changed to be a puppet template and thus be controlled from code with access to config like zulip.conf in supervisor.pp.
  • Whatever is in #/usr/bin/env python controls what happens with other scripts like our cron jobs and Nagios checks.

I think a reasonable next step here might be to add a config option to /etc/zulip/zulip.conf that these things can check for which Python version to run, and use that to control the first 3 places (the last seems hard to deal with!). Another reasonable next step would be to just make a PR that hardcodes all these to use Python3 and test that out in a place like chat.zulip.org to confirm it works.

@timabbott timabbott modified the milestone: Zulip roadmap, Old roadmap Nov 18, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment