Last update was over 2 months ago

[ushuz]

Today is 2016-02-23, the wall was updated at 2015-12-14, it's over 2 months ago. Maybe scheduled tasks went wrong?

https://python3wos.appspot.com/

[tuxayo]

+1 I'm eager to know the migration progress of the ecosystem to Python 3

[ztane]

4 months.

[cclauss]

@ubershmekel This update needs to be run... This site now paints a darker picture of Python 3 adoption than it should!!

[ubershmekel]

Sorry I didn't notice. I'll look into it in the next few days.

[ztane]

@ubershmekel a cronjob would be nice :D

[ubershmekel]

@ztane the cronjob was failing. See 3224e8e for more info. I'm not sure what happened in December - if pypi started responding slowly, or if google changed something. If nothing else is broken - we should see the last updated timestamp climb up slowly during the next few days while all of pypi is crawled.

[cclauss]

@ubershmekel Is it working? The timestamp on the website is not yet moving? Is it correct that the main() function of https://github.com/ubershmekel/python3wos/blob/master/pypi_parser.py is never called!?!

[ztane]

OK, now it is updating something, used to be 2015-12-14, now 2015-12-16.

[ubershmekel]

There was another problem with certain packages failing to update and thus blocking the update list.

@cclauss to follow what gets called in this google app engine cron you go to cron.yaml there you see url: /tasks/update which takes you to https://github.com/ubershmekel/python3wos/blob/master/wsgi.app which mentions ('/tasks/update', pypi_cron.CronUpdate),

So this gets called every 3 minutes: https://github.com/ubershmekel/python3wos/blob/master/pypi_cron.py#L95

The main function is just for my local testing.

[ubershmekel]

And now we've hit this gem, don't worry it'll reset after at most 24 hours:

OverQuotaError: The API call datastore_v3.Put() required more quota than is available.

Maybe I should just give up on updating the counts for all of PyPI. I'm trying to track 19K packages just to show the top 200.

[ushuz]

@ubershmekel I guess crawling the top 200 would be enough.

[cclauss]

My quick and dirty script finds 27 modules in the top 200 scripts still do not support Python3. So 86.5% support Python3 and that is without removing certain modules as the WOS script does. Nice progress...

https://github.com/cclauss/uncategorized_hacks/blob/master/pypi_top200.py

27 Python 2 ONLY packages:

Module name                   Latest       Python 3?
===========                   ======       =========
bcdoc                         0.16.0       False  [deprecated: https://github.com/boto/bcdoc]
supervisor                    3.3.0        False
MySQL-python                  1.2.5        False  [mysqlclient supports Python3]
carbon                        0.9.15       False
futures                       3.0.5        False  [built into Python3]
ssl                           1.16         False  [built into Python3]
graphite-web                  0.9.15       False
Fabric                        1.11.1       False
unittest2                     1.1.0        False  [unittest2py3k supports Python3]
ordereddict                   1.1          False  [built into Python3]
ipaddress                     1.0.16       False  [built into Python3]
mozrunner                     6.11         False
moznetwork                    0.27         False
mozdevice                     0.48         False
mozprofile                    0.28         False
mozprocess                    0.22         False
mozfile                       1.2          False
mozinfo                       0.9          False
mozlog                        3.2          False
mozcrash                      0.17         False
suds                          0.4          False  [suds-jurko supports Python3]
python-gflags                 3.0.5        False
ndg-httpsclient               0.4.0        False
thrift                        0.9.3        False
BeautifulSoup                 3.2.1        False  [beautifulsoup4 supports Python3]
requests-oauthlib             0.6.1        False
functools32                   3.2.3-2      False  [built into Python3]

Stats drop to approx. 70% support for Python3 if you look at the top 1000 modules.

[ubershmekel]

I've added a cron to crawl the top packages more frequently. Within 2 days, the chart should fully update daily.

https://github.com/ubershmekel/python3wos/blob/master/cron.yaml#L7

[cclauss]

@ubershmekel I think that we can close this issue. Nice work!

We can also close issue #16

If we implement equivalents like what is in pull request #26 then I believe that we can close issues #13 and #27

[cclauss]

@ushuz Can you please click Close on this issue?

[ushuz]

@cclauss Closed.