Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Server is down #53

Closed
cclauss opened this Issue Apr 21, 2018 · 9 comments

Comments

Projects
None yet
5 participants
@cclauss
Copy link
Contributor

cclauss commented Apr 21, 2018

@ubershmekel

This comment has been minimized.

Copy link
Owner

ubershmekel commented Apr 22, 2018

Thanks for letting me know. App engine dashboard mentions this since 7 days ago:

Fault: <Fault -32500: 'RuntimeError: This API has been removed. Please Use BigQuery instead.'>
at close (/base/alloc/tmpfs/dynamic_runtimes/python27/c5586dbb532f7e5f_unzipped/python27_dist/lib/python2.7/xmlrpclib.py:800)
at __parse_response (/base/data/home/apps/s~python3wos2/1.405580611997320593/pypi_parser.py:107)
at request (/base/data/home/apps/s~python3wos2/1.405580611997320593/pypi_parser.py:100)
at __request (/base/alloc/tmpfs/dynamic_runtimes/python27/c5586dbb532f7e5f_unzipped/python27_dist/lib/python2.7/xmlrpclib.py:1602)
at __call__ (/base/alloc/tmpfs/dynamic_runtimes/python27/c5586dbb532f7e5f_unzipped/python27_dist/lib/python2.7/xmlrpclib.py:1243)
at get_packages (/base/data/home/apps/s~python3wos2/1.405580611997320593/pypi_parser.py:160)
at fetch_and_cache_package_info (/base/data/home/apps/s~python3wos2/1.405580611997320593/pypi_cron.py:31)
at get_packages_list_from_cache_or_pypi (/base/data/home/apps/s~python3wos2/1.405580611997320593/pypi_cron.py:39)
at build_data (/base/data/home/apps/s~python3wos2/1.405580611997320593/main.py:13)
at get_html (/base/data/home/apps/s~python3wos2/1.405580611997320593/main.py:45)
at get (/base/data/home/apps/s~python3wos2/1.405580611997320593/main.py:59)

Also I see PyPI api failures since Feb 16th.

I might just archive the recent snapshot and be done with it.

@cclauss

This comment has been minimized.

Copy link
Contributor Author

cclauss commented Apr 22, 2018

Understood...

All that remained red was Supervisor that is near to their Python 3 release and the Moz tools.

Not bad given that there are 619 days until Python 2 end of life.

from datetime import date ; print((date(2020, 1, 1) - date.today()).days,
                                  'days until Python 2 end of life.')

@edmorley @hugovk

@ubershmekel

This comment has been minimized.

Copy link
Owner

ubershmekel commented Apr 22, 2018

We're back online. Again thank you @cclauss for letting me know.

This is the offending line, PyPI deprecated the function: package_name_downloads = CLIENT.top_packages(HOW_MANY_TO_CHART)

I took some recent download numbers from Big Query https://bigquery.cloud.google.com/table/the-psf:pypi.downloads20180422 using this query:

SELECT file.project, COUNT(*) as download_count
FROM TABLE_DATE_RANGE(
  [the-psf:pypi.downloads],
  TIMESTAMP("2018-04-01"),
  CURRENT_TIMESTAMP()
)
GROUP BY file.project
ORDER BY download_count DESC
LIMIT 200

I then downloaded the JSON, ran the rest of the code using the above data, got the template data that would have been used to render html, saved that, and from now on we're forever using that stale snapshot of data. I added some deprecation and stale data warning to the top of the site.

ubershmekel added a commit that referenced this issue Apr 22, 2018

Recovering from the 500 error
Saving the template data to a file
I built this final data set from Google Big Query
More info at #53
@hugovk

This comment has been minimized.

Copy link
Contributor

hugovk commented Apr 22, 2018

For fresh data, you could do something like meshy/pythonwheels#100.

@hugovk

This comment has been minimized.

Copy link
Contributor

hugovk commented Apr 22, 2018

... but I think it's fine to say "mission over" and keep it as is.

Thank you very much for making this in the first place and helping to get to Python 3!

@brainwane

This comment has been minimized.

Copy link

brainwane commented Apr 22, 2018

I wonder whether the libraries.io API https://libraries.io/api would be a useful substitute?

@brainwane

This comment has been minimized.

@cclauss cclauss closed this Apr 23, 2018

@cclauss

This comment has been minimized.

Copy link
Contributor Author

cclauss commented May 11, 2018

@ubershmekel Can you please regenerate the page? I believe there has been some improvement.

@dmsquires

This comment has been minimized.

Copy link

dmsquires commented May 22, 2018

I would rather create a "wall of shame" for still-popular modules who haven't adopted a proper python3 path versus starting this up again.

Great job all around on this, I visited this site monthly for ages. It solidified my push to use python3 exclusively in all my projects a long time ago.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.