Dependency resolution for azure-core failing due to transitive depdendencies

[tanertopal]

Describe the bug
I am using Bazel with rules_python to generate a lock file for my dependencies. In the process of generating the lock file, I am encountering which you can see below.

farm-haystack dependents on azure-core<1.23 and some transitive dependencies on azure-core>=1.23.

Is this a known issue or does anyone know a fix?

Error message

Could not find a version that matches azure-core<1.23,<2.0.0,>=1.13.0,>=1.24.0 (from farm-haystack[crawler,preprocessing]==1.6.0->-r third_party/py/requirements.in.txt (line 6))
Tried: 1.0.0, 1.0.0, 1.1.0, 1.1.0, 1.1.1, 1.1.1, 1.2.0, 1.2.0, 1.2.1, 1.2.1, 1.2.2, 1.2.2, 1.3.0, 1.3.0, 1.4.0, 1.4.0, 1.5.0, 1.5.0, 1.6.0, 1.6.0, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.8.1, 1.8.1, 1.8.2, 1.8.2, 1.9.0, 1.9.0, 1.10.0, 1.10.0, 1.11.0, 1.11.0, 1.12.0, 1.12.0, 1.13.0, 1.13.0, 1.14.0, 1.14.0, 1.15.0, 1.15.0, 1.16.0, 1.16.0, 1.17.0, 1.17.0, 1.18.0, 1.18.0, 1.19.0, 1.19.0, 1.19.1, 1.19.1, 1.20.0, 1.20.0, 1.20.1, 1.20.1, 1.21.0, 1.21.0, 1.21.1, 1.21.1, 1.22.0, 1.22.0, 1.22.1, 1.22.1, 1.23.0, 1.23.0, 1.23.1, 1.23.1, 1.24.0, 1.24.0, 1.24.1, 1.24.1, 1.24.2, 1.24.2, 1.25.0, 1.25.0
Skipped pre-versions: 1.0.0b1, 1.0.0b1, 1.0.0b2, 1.0.0b2, 1.0.0b3, 1.0.0b3, 1.0.0b4, 1.0.0b4, 1.12.0b1, 1.12.0b1, 1.13.0b1, 1.13.0b1, 1.14.0b1, 1.14.0b1, 1.15.0b1, 1.15.0b1
There are incompatible versions in the resolved dependencies:
  azure-core>=1.24.0 (from msrest==0.7.1->azure-ai-formrecognizer==3.2.0b2->farm-haystack[crawler,preprocessing]==1.6.0->-r third_party/py/requirements.in.txt (line 6))
  azure-core<2.0.0,>=1.13.0 (from azure-ai-formrecognizer==3.2.0b2->farm-haystack[crawler,preprocessing]==1.6.0->-r third_party/py/requirements.in.txt (line 6))
  azure-core<1.23 (from farm-haystack[crawler,preprocessing]==1.6.0->-r third_party/py/requirements.in.txt (line 6))

Expected behavior

Transitive and direct dependencies should resolve correctly.

Additional context

• Using Bazel with rules_python

To Reproduce

1. Create a requirements.txt with the content:

farm-haystack[crawler,preprocessing]==1.6.0

2. Run

bazel run //third_party/py:requirements.update

Would it help if I created an example project?

[TuanaCelik]

@ZanSara could you help out here? This might be a known issue but I am unsure 😊

[ZanSara]

Hello @tanertopal, thank you for this report. So, we had issues in the past that lead us to the inclusion of a pin to azure-core<1.23 in the core dependencies. You can see the reason here:

haystack/setup.cfg

Lines 71 to 77 in 408d8e6

	# azure-core is a dependency of azure-ai-formrecognizer
	# In order to stop malicious pip backtracking during pip install farm-haystack[all] documented in https://github.com/deepset-ai/haystack/issues/2280
	# we have to resolve a dependency version conflict ourself.
	# azure-core>=1.23 conflicts with pydoc-markdown's dependency on databind>=1.5.0 which itself requires typing-extensions<4.0.0
	# azure-core>=1.23 needs typing-extensions>=4.0.1
	# pip unfortunately backtracks into the databind direction ultimately getting lost.
	azure-core<1.23

However, re-reading the reasoning of the comment and your issue, I realize the pin might be better added to the dev dependency group, where pydoc-markdown is required, rather than in the core dependencies list. Hopefully this will not cause mismatches between the development and the minimal dependency list: I will open a separate issue as a reminder for us to monitor the situation closely.

TL;DR: I will open a PR shortly and ask you to test if you can lock your dependencies using that branch.

[ZanSara]

Hey @tanertopal, please checkout #3022 and let me know if it helps. If we merge this PR this week it will be included in the next Haystack release on Monday 😊

[tanertopal]

@ZanSara, I tried it out and its works and resolves my issue! Thank you for the fast turnaround!

[ZanSara]

Great, thanks for checking so quickly! It will be in master soon.