Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

package exist on pypi but not sync #1529

Closed
89ao opened this issue Aug 28, 2023 · 8 comments
Closed

package exist on pypi but not sync #1529

89ao opened this issue Aug 28, 2023 · 8 comments

Comments

@89ao
Copy link
Contributor

89ao commented Aug 28, 2023

as log below:

[root@softsource-agent-pypi-0 /data]# bandersnatch sync simple
2023-08-28 14:24:55,082 bandersnatch: INFO Selected storage backend: filesystem
2023-08-28 14:24:55,082 bandersnatch: INFO Selected compare method: stat
2023-08-28 14:24:55,082 bandersnatch: INFO Selected alternative download mirror https://mirrors.tuna.tsinghua.edu.cn
2023-08-28 14:24:55,236 bandersnatch: INFO Initialized project plugin blocklist_project, filtering ['pyagrum-nightly', 'tf-nightly', 'tf-nightly-cpu', 'tf-nightly-gpu', 'uselesscapitalquiz', 'appium', 'tensorflow-io-nightly']
2023-08-28 14:24:55,241 bandersnatch: INFO Initialized regex release plugin with [re.compile('.+-nightly.*')]
2023-08-28 14:24:55,258 bandersnatch: INFO Initialized release plugin blocklist_release, filtering [<Requirement('pyagrum-nightly')>, <Requirement('tf-nightly-cpu')>, <Requirement('appium')>, <Requirement('uselesscapitalquiz')>, <Requirement('tf-nightly-gpu')>, <Requirement('tf-nightly')>, <Requirement('tensorflow-io-nightly')>]
2023-08-28 14:24:55,308 bandersnatch.mirror: INFO Syncing with https://pypi.org/.
2023-08-28 14:24:55,309 bandersnatch.mirror: INFO No metadata filters are enabled. Skipping metadata filtering
2023-08-28 14:24:55,309 bandersnatch.mirror: INFO No release file filters are enabled. Skipping release file filtering
2023-08-28 14:24:55,309 bandersnatch.package: INFO Fetching metadata for package: simple (serial 0)
2023-08-28 14:24:57,399 bandersnatch.package: INFO simple no longer exists on PyPI
2023-08-28 14:24:57,399 bandersnatch.simple: INFO Generating global index page.

however，"simple" package exist on pypi :https://pypi.org/project/simple/
@89ao
Copy link
Contributor Author

89ao commented Aug 28, 2023

same issue happens on flask-ipfilter,Prix-Carburant-FR-Client packages

@cooperlees
Copy link
Contributor

as log below:

[root@softsource-agent-pypi-0 /data]# bandersnatch sync simple
2023-08-28 14:24:55,082 bandersnatch: INFO Selected storage backend: filesystem
2023-08-28 14:24:55,082 bandersnatch: INFO Selected compare method: stat
2023-08-28 14:24:55,082 bandersnatch: INFO Selected alternative download mirror https://mirrors.tuna.tsinghua.edu.cn
2023-08-28 14:24:55,236 bandersnatch: INFO Initialized project plugin blocklist_project, filtering ['pyagrum-nightly', 'tf-nightly', 'tf-nightly-cpu', 'tf-nightly-gpu', 'uselesscapitalquiz', 'appium', 'tensorflow-io-nightly']
2023-08-28 14:24:55,241 bandersnatch: INFO Initialized regex release plugin with [re.compile('.+-nightly.*')]
2023-08-28 14:24:55,258 bandersnatch: INFO Initialized release plugin blocklist_release, filtering [<Requirement('pyagrum-nightly')>, <Requirement('tf-nightly-cpu')>, <Requirement('appium')>, <Requirement('uselesscapitalquiz')>, <Requirement('tf-nightly-gpu')>, <Requirement('tf-nightly')>, <Requirement('tensorflow-io-nightly')>]
2023-08-28 14:24:55,308 bandersnatch.mirror: INFO Syncing with https://pypi.org/.
2023-08-28 14:24:55,309 bandersnatch.mirror: INFO No metadata filters are enabled. Skipping metadata filtering
2023-08-28 14:24:55,309 bandersnatch.mirror: INFO No release file filters are enabled. Skipping release file filtering
2023-08-28 14:24:55,309 bandersnatch.package: INFO Fetching metadata for package: simple (serial 0)
2023-08-28 14:24:57,399 bandersnatch.package: INFO simple no longer exists on PyPI
2023-08-28 14:24:57,399 bandersnatch.simple: INFO Generating global index page.

however，"simple" package exist on pypi :https://pypi.org/project/simple/

Simple's Simple API page has no packages, so we can not mirror non existent packages: https://pypi.org/simple/simple/

It could be a bug with PyPI maybe?

@89ao
Copy link
Contributor Author

89ao commented Aug 31, 2023

I agree， seems that the "simple" package has a version 0.1.1 but doesn't have a package,interesting.
image

@89ao 89ao closed this as completed Aug 31, 2023
@89ao
Copy link
Contributor Author

89ao commented Aug 31, 2023

how about this one:adafruit-python-dht
https://pypi.org/project/Adafruit_Python_DHT/ http returns 200
yet "bandersnatch sync adafruit_python_dht" log below :

[root@softsource-agent-pypi-0 /data]# bandersnatch sync adafruit_python_dht
2023-08-31 10:27:19,106 bandersnatch: INFO Selected storage backend: filesystem
2023-08-31 10:27:19,107 bandersnatch: INFO Selected compare method: stat
2023-08-31 10:27:19,107 bandersnatch: INFO Selected alternative download mirror https://mirrors.tuna.tsinghua.edu.cn
2023-08-31 10:27:19,242 bandersnatch: INFO Initialized project plugin blocklist_project, filtering ['tf-nightly-cpu', 'appium', 'tf-nightly-gpu', 'tensorflow-io-nightly', 'uselesscapitalquiz', 'pyagrum-nightly', 'tf-nightly']
2023-08-31 10:27:19,246 bandersnatch: INFO Initialized regex release plugin with [re.compile('.+-nightly.*')]
2023-08-31 10:27:19,260 bandersnatch: INFO Initialized release plugin blocklist_release, filtering [<Requirement('appium')>, <Requirement('uselesscapitalquiz')>, <Requirement('tf-nightly-gpu')>, <Requirement('pyagrum-nightly')>, <Requirement('tf-nightly')>, <Requirement('tf-nightly-cpu')>, <Requirement('tensorflow-io-nightly')>]
2023-08-31 10:27:19,307 bandersnatch.mirror: INFO Syncing with https://pypi.org/.
2023-08-31 10:27:19,307 bandersnatch.mirror: INFO No metadata filters are enabled. Skipping metadata filtering
2023-08-31 10:27:19,307 bandersnatch.mirror: INFO No release file filters are enabled. Skipping release file filtering
2023-08-31 10:27:19,307 bandersnatch.package: INFO Fetching metadata for package: adafruit-python-dht (serial 0)
**2023-08-31 10:27:20,931 bandersnatch.package: INFO adafruit-python-dht no longer exists on PyPI**
2023-08-31 10:27:20,931 bandersnatch.simple: INFO Generating global index page.

Seems like there's a bit of an issue with the conversion of the characters '-' and '_' ？

@89ao 89ao reopened this Aug 31, 2023
@89ao
Copy link
Contributor Author

89ao commented Sep 5, 2023

one more case :
https://pypi.org/project/thrift-sasl/
Pypi.org returns 200 yet bandersnatch cannot sync it locally.

[root@softsource-agent-pypi-0 /data]# bandersnatch sync thrift-sasl 
2023-09-05 20:06:52,749 bandersnatch: INFO Selected storage backend: filesystem
2023-09-05 20:06:52,750 bandersnatch: INFO Selected compare method: stat
2023-09-05 20:06:52,750 bandersnatch: INFO Selected alternative download mirror https://mirrors.tuna.tsinghua.edu.cn
2023-09-05 20:06:52,929 bandersnatch: INFO Initialized project plugin blocklist_project, filtering ['tf-nightly', 'tensorflow-io-nightly', 'tf-nightly-cpu', 'appium', 'tf-nightly-gpu', 'uselesscapitalquiz', 'pyagrum-nightly']
2023-09-05 20:06:52,935 bandersnatch: INFO Initialized regex release plugin with [re.compile('.+-nightly.*')]
2023-09-05 20:06:52,955 bandersnatch: INFO Initialized release plugin blocklist_release, filtering [<Requirement('uselesscapitalquiz')>, <Requirement('appium')>, <Requirement('tf-nightly')>, <Requirement('tf-nightly-cpu')>, <Requirement('tensorflow-io-nightly')>, <Requirement('tf-nightly-gpu')>, <Requirement('pyagrum-nightly')>]
2023-09-05 20:06:53,011 bandersnatch.mirror: INFO Syncing with https://pypi.org/.
2023-09-05 20:06:53,012 bandersnatch.mirror: INFO No metadata filters are enabled. Skipping metadata filtering
2023-09-05 20:06:53,012 bandersnatch.mirror: INFO No release file filters are enabled. Skipping release file filtering
2023-09-05 20:06:53,012 bandersnatch.package: INFO Fetching metadata for package: thrift-sasl (serial 0)
===2023-09-05 20:06:54,631 bandersnatch.package: INFO thrift-sasl no longer exists on PyPI =====
...

@cooperlees
Copy link
Contributor

cooperlees commented Sep 6, 2023 •
edited
Loading

how about this one:adafruit-python-dht https://pypi.org/project/Adafruit_Python_DHT/ http returns 200 yet "bandersnatch sync adafruit_python_dht" log below :

Seems like there's a bit of an issue with the conversion of the characters '-' and '_' ？

Worked for me installing bandersnatch from master and running on my laptop:

crl-m1:bandersnatch cooper$ /tmp/tb/bin/bandersnatch sync adafruit_python_dht
2023-09-05 21:21:11,292 INFO: Selected storage backend: filesystem (configuration.py:132)
2023-09-05 21:21:11,292 INFO: Selected compare method: hash (configuration.py:180)
2023-09-05 21:21:11,323 INFO: Setting up mirror directory: /tmp/pypi/web/simple (mirror.py:505)
2023-09-05 21:21:11,323 INFO: Setting up mirror directory: /tmp/pypi/web/packages (mirror.py:505)
2023-09-05 21:21:11,323 INFO: Setting up mirror directory: /tmp/pypi/web/local-stats/days (mirror.py:505)
2023-09-05 21:21:11,323 INFO: Setting up mirror directory: /tmp/pypi/web/json (mirror.py:505)
2023-09-05 21:21:11,323 INFO: Setting up mirror directory: /tmp/pypi/web/pypi (mirror.py:505)
2023-09-05 21:21:11,323 INFO: Generation file missing. Reinitialising status files. (mirror.py:545)
2023-09-05 21:21:11,325 INFO: Status file /tmp/pypi/status missing. Starting over. (mirror.py:567)
2023-09-05 21:21:11,325 INFO: Syncing with https://pypi.org. (mirror.py:58)
2023-09-05 21:21:11,325 INFO: No metadata filters are enabled. Skipping metadata filtering (mirror.py:77)
2023-09-05 21:21:11,325 INFO: No release filters are enabled. Skipping release filtering (mirror.py:79)
2023-09-05 21:21:11,325 INFO: No release file filters are enabled. Skipping release file filtering (mirror.py:81)
2023-09-05 21:21:11,325 INFO: Fetching metadata for package: adafruit-python-dht (serial 0) (package.py:58)
2023-09-05 21:21:11,511 INFO: Downloading: https://files.pythonhosted.org/packages/2c/ed/482d4e0aaf7221996db3173044de4d09fc2c2a7e3589050ad09b6d24758c/Adafruit_Python_DHT-1.1.0.tar.gz (mirror.py:876)
2023-09-05 21:21:11,736 INFO: Downloading: https://files.pythonhosted.org/packages/09/99/fd16d79b5ae210aa10c96fa60aeb85a13b87bd8aafb2d7b4b7e0b365ad32/Adafruit_Python_DHT-1.1.1.tar.gz (mirror.py:876)
2023-09-05 21:21:11,767 INFO: Downloading: https://files.pythonhosted.org/packages/73/f3/b5e473c3d058870b04b401210139f9bdba85cc2dbfbfb2b57a6b6ffb2318/Adafruit_Python_DHT-1.1.2.tar.gz (mirror.py:876)
2023-09-05 21:21:11,859 INFO: Downloading: https://files.pythonhosted.org/packages/52/aa/418b82e36fc57316137679c2a12b8bae89ab73f211cbca91a3275e0b5fc2/Adafruit_Python_DHT-1.3.2.tar.gz (mirror.py:876)
2023-09-05 21:21:11,948 INFO: Downloading: https://files.pythonhosted.org/packages/b0/76/bdce34fd6e58cd91a6e4fc39a06a6fd362fd0cdec71fb77ad1c6a9f0c545/Adafruit_Python_DHT-1.3.3-py3.5-linux-x86_64.egg (mirror.py:876)
2023-09-05 21:21:12,006 INFO: Downloading: https://files.pythonhosted.org/packages/aa/9d/21e7f902e693c65a2a6c46ca327b8a15751fc04c3cf0ecb5e692540f6ab8/Adafruit_Python_DHT-1.3.3.tar.gz (mirror.py:876)
2023-09-05 21:21:12,096 INFO: Downloading: https://files.pythonhosted.org/packages/d6/85/af539216f80d10e917eca17c52085963765646a3500a9dd11ed8e9d9b7df/Adafruit_Python_DHT-1.3.4.tar.gz (mirror.py:876)
2023-09-05 21:21:12,185 INFO: Downloading: https://files.pythonhosted.org/packages/e9/e5/8261ce3480ad08567123286501628bfe1452b13bc4f20e98b46e456b060b/Adafruit_Python_DHT-1.4.0.tar.gz (mirror.py:876)
2023-09-05 21:21:12,279 INFO: Storing index page(s): adafruit-python-dht - in /tmp/pypi/web/simple/a/adafruit-python-dht (mirror.py:699)
2023-09-05 21:21:12,282 INFO: Generating global index page. (simple.py:259)
2023-09-05 21:21:12,385 INFO: 1 packages had changes (mirror.py:991)
2023-09-05 21:21:12,386 INFO: Writing diff file to mirrored-files (mirror.py:1001)

So did thrift-sasl:

crl-m1:bandersnatch cooper$ /tmp/tb/bin/bandersnatch sync thrift-sasl
2023-09-05 21:24:23,109 INFO: Selected storage backend: filesystem (configuration.py:132)
2023-09-05 21:24:23,110 INFO: Selected compare method: hash (configuration.py:180)
2023-09-05 21:24:23,144 ERROR: Removing inconsistent todo list. (mirror.py:401)
2023-09-05 21:24:23,144 INFO: Removing file: /tmp/pypi/todo (filesystem.py:188)
2023-09-05 21:24:23,145 INFO: Status file /tmp/pypi/status missing. Starting over. (mirror.py:567)
2023-09-05 21:24:23,145 INFO: Syncing with https://pypi.org. (mirror.py:58)
2023-09-05 21:24:23,145 INFO: No metadata filters are enabled. Skipping metadata filtering (mirror.py:77)
2023-09-05 21:24:23,145 INFO: No release filters are enabled. Skipping release filtering (mirror.py:79)
2023-09-05 21:24:23,145 INFO: No release file filters are enabled. Skipping release file filtering (mirror.py:81)
2023-09-05 21:24:23,145 INFO: Fetching metadata for package: thrift-sasl (serial 0) (package.py:58)
2023-09-05 21:24:23,319 INFO: Downloading: https://files.pythonhosted.org/packages/ca/24/120d2f48efc65204d859081e4eb0a18695e7531c4575bb136a25746a1381/thrift_sasl-0.1.0-py2.7.egg (mirror.py:876)
2023-09-05 21:24:23,453 INFO: Downloading: https://files.pythonhosted.org/packages/5b/e4/371dbc680035ec788d40216b9daf0526a79b329070ba7272d4e3fc4d2c4e/thrift_sasl-0.1.0.tar.gz (mirror.py:876)
2023-09-05 21:24:23,550 INFO: Downloading: https://files.pythonhosted.org/packages/0f/d6/ac7fcc6dbc1475aa3b23dbee126788d59b487054e3f458c446f3ce15d1d7/thrift_sasl-0.2.0.tar.gz (mirror.py:876)
2023-09-05 21:24:23,641 INFO: Downloading: https://files.pythonhosted.org/packages/80/36/16dfe92d32df63cc2b7b7be8d0e4a736617b7e52daaa7f83ae386a89d179/thrift_sasl-0.2.1.tar.gz (mirror.py:876)
2023-09-05 21:24:23,676 INFO: Downloading: https://files.pythonhosted.org/packages/50/fe/89cbc910809e3757c762f56ee190ca39e0f28b7ea451835232c0c988d706/thrift_sasl-0.3.0.tar.gz (mirror.py:876)
2023-09-05 21:24:23,702 INFO: Downloading: https://files.pythonhosted.org/packages/1b/df/a744ed6929808c52d1ee0648557f1a1430400d364f0ffdd9ea28d7127579/thrift_sasl-0.4.1.tar.gz (mirror.py:876)
2023-09-05 21:24:23,729 INFO: Downloading: https://files.pythonhosted.org/packages/73/d3/588654faef5511afadc1a091d32fcdbb24ae5f2d90b380874aee68a717f9/thrift_sasl-0.4.2.tar.gz (mirror.py:876)
2023-09-05 21:24:23,756 INFO: Downloading: https://files.pythonhosted.org/packages/c3/9e/636c24ce1c0d46ce3020c5836c5a375d8e862fa81a240e0e352cc991dcf8/thrift_sasl-0.4.3-py2.py3-none-any.whl (mirror.py:876)
2023-09-05 21:24:23,785 INFO: Downloading: https://files.pythonhosted.org/packages/fa/fb/5d9ba73be60b32f6fda080a5556cff25e15daf33bafe043c226404b39b25/thrift_sasl-0.4.3.tar.gz (mirror.py:876)
2023-09-05 21:24:23,812 INFO: Downloading: https://files.pythonhosted.org/packages/f2/08/912c32ecbe6e5b8ed4eec93bee3e0b9cce8ca335576194bbd07c1931be46/thrift_sasl-0.4.3a1-py2.py3-none-any.whl (mirror.py:876)
2023-09-05 21:24:23,908 INFO: Downloading: https://files.pythonhosted.org/packages/42/cb/ee76ab613e3b4e01dfb6044558b8e9838f9f97e0a37cc2e0ecffc6a7c606/thrift_sasl-0.4.3a1.tar.gz (mirror.py:876)
2023-09-05 21:24:23,996 INFO: Downloading: https://files.pythonhosted.org/packages/d0/0a/92e961e6a30222ef1775d15a0c11f89309101f6e98e4ba709462760f8a40/thrift_sasl-0.4.3a2-py2.py3-none-any.whl (mirror.py:876)
2023-09-05 21:24:24,033 INFO: Downloading: https://files.pythonhosted.org/packages/39/ac/713d03a8d6d73e5d35e7ae06ad8e10543c7adb64329eee8b1570af38f583/thrift_sasl-0.4.3a2.tar.gz (mirror.py:876)
2023-09-05 21:24:24,126 INFO: Downloading: https://files.pythonhosted.org/packages/61/fd/a24996119bd346ca06a182135c1894609ccb5d35f5e0e81be83877fc4005/thrift_sasl-0.4a1.tar.gz (mirror.py:876)
2023-09-05 21:24:24,226 INFO: Storing index page(s): thrift-sasl - in /tmp/pypi/web/simple/t/thrift-sasl (mirror.py:699)
2023-09-05 21:24:24,230 INFO: Generating global index page. (simple.py:259)
2023-09-05 21:24:24,333 INFO: 1 packages had changes (mirror.py:991)
2023-09-05 21:24:24,333 INFO: Writing diff file to mirrored-files (mirror.py:1001)

The config was:

crl-m1:bandersnatch cooper$ cat /etc/bandersnatch.conf
[mirror]
directory = /tmp/pypi
json = true
cleanup = true
master = https://pypi.org
timeout = 60
global-timeout = 18000
workers = 3
hash-index = true
stop-on-error = true
storage-backend = filesystem
verifiers = 3
keep_index_versions = 2
compare-method = hash

So don't know what to fix here sorry ...

@89ao
Copy link
Contributor Author

89ao commented Sep 6, 2023

tks @cooperlees ，maybe I need to update to a new version and have a try!

@cooperlees
Copy link
Contributor

Looking again at your logs, either the alternative mirror you're using is broken or we have a bug when using an alternative mirror:

bandersnatch: INFO Selected alternative download mirror https://mirrors.tuna.tsinghua.edu.cn

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cooperlees @89ao