Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

If using multiple package indices, an unresolved host error prevents the subsequent indices from being tried #1945

Closed
1 task done
taranlu-houzz opened this issue May 22, 2023 · 3 comments · Fixed by #1949 · May be fixed by frostming/unearth#54
Closed
1 task done

If using multiple package indices, an unresolved host error prevents the subsequent indices from being tried #1945

taranlu-houzz opened this issue May 22, 2023 · 3 comments · Fixed by #1949 · May be fixed by frostming/unearth#54
Assignees
@frostming
Labels
🐛 bug Something isn't working

Comments

@taranlu-houzz
Copy link

  • I have searched the issue tracker and believe that this is not a duplicate.

Make sure you run commands with -v flag before pasting the output.

Steps to reproduce

I have a situation where the DNS for an internal pypi index is different between my local laptop and the ci node we use to build packages (I think it is probably the same EC2 instance, but the DNS is just different). This makes dev/ci a bit challenging, since pdm fails with an error on pdm install if the dns for a source cannot be resolved, even if multiple sources have been specified. Here is an example of the setup that I am trying:

[tool.pdm.resolution]
respect-source-order = true

[[tool.pdm.source]]
name = "ci"
url = "http://<name 1>.com:8081"
verify_ssl = false

[[tool.pdm.source]]
name = "local"
url = "http://<name 2>:8081"
verify_ssl = false

Each source works individually for either CI on Jenkins or locally on my laptop.

Actual behavior

pdm fails due to a exception raised when it is unable to resolve the DNS for one of the sources:

requests.exceptions.ConnectionError: HTTPConnectionPool(host='<name 1>', port=8081): Max retries exceeded with url: /<my package name>/ (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x1062abd30>: Failed to resolve '<name 1>' ([Errno 8] nodename nor servname provided, or not known)")
  • I replaced the actual host name and package name for this issue manually with e.g. <name 1>.

Expected behavior

Instead of failing with an exception, it seems like it would be better to catch the error, log it, and then move on to the next source (if available).

Environment Information

❯ pdm info && pdm info --env
PDM version:
  2.6.1
Python Interpreter:
  /Users/<path to project dir>/<package name>/.venv/bin/python (3.7)
Project Root:
  /Users/<path to project dir>
Local Packages:

{
  "implementation_name": "cpython",
  "implementation_version": "3.7.16",
  "os_name": "posix",
  "platform_machine": "arm64",
  "platform_release": "22.4.0",
  "platform_system": "Darwin",
  "platform_version": "Darwin Kernel Version 22.4.0: Mon Mar  6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000",
  "python_full_version": "3.7.16",
  "platform_python_implementation": "CPython",
  "python_version": "3.7",
  "sys_platform": "darwin"
}
@taranlu-houzz taranlu-houzz added the 🐛 bug Something isn't working label May 22, 2023
taranlu-houzz added a commit to taranlu-houzz/unearth that referenced this issue May 22, 2023
@taranlu-houzz
Add try/catch to catch ConnectionErrors that can occur when using req…
d36b467 
…uests to get package url

- Specificall, this resolves pdm-project/pdm#1945 which was caused by an unknown host name due to not having a DNS entry.
@taranlu-houzz taranlu-houzz mentioned this issue May 22, 2023
Draft
@frostming
Copy link
Collaborator

I don't agree, LinkCollectionError will be silent unless -v is passed, which makes you never have a chance to spot the error.

@taranlu-houzz
Copy link
Author

That is a good point, but that means that there isn't really any way to resolve the situation that I have run into. Because the DNS is different locally vs ci, the project will have to choose one or the other for the source, since having both will break things. I am trying to see if we can get the DNS issue consistent, but it seems to me that others might run into a similar issue...

Would it be possible to add a parameter for sources that allows them to be skipped if they fail?

@taranlu-houzz
Copy link
Author

taranlu-houzz commented May 23, 2023 •
edited
Loading

@frostming So, in order to get around this, I figured that I could use the global pdm config to set the repository, and just have different config in ci vs local. This works, but I did notice that it seems that pdm config --global pypi.<my pypi index name>.verify_ssl False results in a quoted string "False" in the pdm/config.toml file (instead of the needed false for a valid toml bool), which then fails to be considered a valid, trusted_host here:

pdm/src/pdm/utils.py

Line 64 in 1638e62

if host not in trusted_hosts and source.verify_ssl is False:

So, basically, it seems that it is not possible to set a true bool value using the pdm config command currently.

@frostming frostming self-assigned this May 24, 2023
frostming added a commit that referenced this issue May 24, 2023
@frostming
fix: convert to boolean when setting verify_ssl for custom indexes
b6309b6 
Fixes #1945

Signed-off-by: Frost Ming <me@frostming.com>
@frostming frostming mentioned this issue May 24, 2023
Merged
2 tasks
frostming added a commit that referenced this issue May 24, 2023
@frostming
fix: convert to boolean when setting verify_ssl for custom indexes (#…
4ab8292 
…1949)

Fixes #1945

Signed-off-by: Frost Ming <me@frostming.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@frostming frostming

Labels
🐛 bug Something isn't working
Projects
None yet
Milestone
No milestone
2 participants
@frostming @taranlu-houzz