Outdated database

[jfmcarreira]

I was trying to install guidata package and noticed that the database is outdated.

This is an issue cause either the older versions are removed online or they moved somewhere else.

How is the database generated? Automatically or you need to updated by hand? If it is automatic could not be possible to g-sorcery do it when requested?

[hoefling]

Take a look at gs-pypi-generate-db. After db is generated, create a git repo in db dir and adapt /etc/g-sorcery/gs-pypi.json to include your custom db:

"repositories": {
    "pypi": {
      "repo_uri": "https://pypi.python.org/",
      "sync_method": "git",
      "db_uri": "file:///path/to/db",
      "branch": "master"
    }
  }
...

[heroxbd]

Related bug report: https://bugs.gentoo.org/607666

[weedy]

Is this even possible anymore?

# gs-pypi-generate-db test
--2018-08-13 17:20:26--  https://pypi.python.org/pypi?%3Aaction=index
Resolving pypi.python.org... 151.101.124.223, 2a04:4e42:1e::223
Connecting to pypi.python.org|151.101.124.223|:443... connected.
HTTP request sent, awaiting response... 301 Redirect to Primary Domain
Location: https://pypi.org/pypi?%3Aaction=index [following]
--2018-08-13 17:20:26--  https://pypi.org/pypi?%3Aaction=index
Resolving pypi.org... 151.101.64.223, 151.101.192.223, 151.101.128.223, ...
Connecting to pypi.org|151.101.64.223|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://pypi.org/pypi/?%3Aaction=index [following]
--2018-08-13 17:20:26--  https://pypi.org/pypi/?%3Aaction=index
Reusing existing connection to pypi.org:443.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://pypi.org/ [following]
--2018-08-13 17:20:26--  https://pypi.org/
Reusing existing connection to pypi.org:443.
HTTP request sent, awaiting response... 200 OK
Length: 17271 (17K) [text/html]
Saving to: ‘/tmp/tmp0zlfwza7/packages’

/tmp/tmp0zlfwza7/packages           100%[=================================================================>]  16.87K  --.-KB/s    in 0.02s   

2018-08-13 17:20:26 (1.04 MB/s) - ‘/tmp/tmp0zlfwza7/packages’ saved [17271/17271]

/usr/lib64/python3.5/site-packages/bs4/__init__.py:181: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

The code that caused this warning is on line 18 of the file /usr/lib/python-exec/python3.5/gs-pypi-generate-db. To get rid of this warning, change code that looks like this:

 BeautifulSoup(YOUR_MARKUP})

to this:

 BeautifulSoup(YOUR_MARKUP, "lxml")

  markup_type=markup_type))
Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.5/gs-pypi-generate-db", line 18, in <module>
    sys.exit(gs_pypi_generate_db.main())
  File "/usr/lib64/python3.5/site-packages/gs_pypi/gs_pypi_generate_db.py", line 68, in main
    common_config=config["common_config"])
  File "/usr/lib64/python3.5/site-packages/g_sorcery/package_db.py", line 563, in __call__
    self.generate_tree(pkg_db, common_config, config)
  File "/usr/lib64/python3.5/site-packages/g_sorcery/package_db.py", line 577, in generate_tree
    data = self.download_data(common_config, config)
  File "/usr/lib64/python3.5/site-packages/g_sorcery/package_db.py", line 595, in download_data
    self.process_uri(uri, data)
  File "/usr/lib64/python3.5/site-packages/g_sorcery/package_db.py", line 606, in process_uri
    data.update(load_remote_file(**uri))
  File "/usr/lib64/python3.5/site-packages/g_sorcery/fileutils.py", line 301, in load_remote_file
    open_file=open_file, open_mode=open_mode))
  File "/usr/lib64/python3.5/site-packages/g_sorcery/fileutils.py", line 258, in _call_parser
    data = parser(f)
  File "/usr/lib64/python3.5/site-packages/gs_pypi/pypi_db.py", line 62, in parse_data
    for entry in packages.find_all("tr")[1:last]:
AttributeError: 'NoneType' object has no attribute 'find_all'