Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: rescue docarray in setup #4203

Merged
merged 1 commit into from
Jan 24, 2022
Merged

fix: rescue docarray in setup #4203

merged 1 commit into from
Jan 24, 2022

Conversation

jacobowitz
Copy link
Contributor

This closes #4194

Upgrading from Jina 2.x to 3.x is broken because pip will install first the new docarray pip package and then remove the old docarray package (which was part of jina repo). Because both are installed in the same folder, this will leave the new docarray package in some invalid state.
This change to the setup introduces a hack to rescue those cases: My code is checking if docarray can be imported correctly (by checking access to the version, this fails for master now). If this fails docarray is reinstalled via pip.

In non upgrading cases we will not enter the exception handler and my hack will not do anything (when installing jina in a new venv).

@github-actions github-actions bot added size/S area/setup This issue/PR affects setting up Jina labels Jan 21, 2022
@codecov
Copy link

codecov bot commented Jan 21, 2022

Codecov Report

Merging #4203 (20d28a8) into master (4db2069) will increase coverage by 1.15%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #4203      +/-   ##
==========================================
+ Coverage   85.72%   86.88%   +1.15%     
==========================================
  Files         114      114              
  Lines        8304     8304              
==========================================
+ Hits         7119     7215      +96     
+ Misses       1185     1089      -96     
Flag Coverage Δ
daemon 42.49% <ø> (+0.01%) ⬆️
jina 86.63% <ø> (+2.09%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
jina/jaml/__init__.py 95.47% <0.00%> (+0.41%) ⬆️
jina/serve/runtimes/gateway/http/app.py 92.13% <0.00%> (+1.12%) ⬆️
jina/helper.py 81.50% <0.00%> (+1.19%) ⬆️
jina/jaml/parsers/flow/v1.py 98.24% <0.00%> (+1.75%) ⬆️
jina/clients/base/http.py 96.07% <0.00%> (+1.96%) ⬆️
jina/orchestrate/peas/jinad.py 92.70% <0.00%> (+2.18%) ⬆️
jina/jaml/helper.py 85.36% <0.00%> (+2.43%) ⬆️
jina/orchestrate/peas/__init__.py 89.37% <0.00%> (+2.50%) ⬆️
jina/orchestrate/peas/helper.py 88.57% <0.00%> (+2.85%) ⬆️
jina/clients/__init__.py 90.90% <0.00%> (+6.06%) ⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4db2069...20d28a8. Read the comment docs.

@github-actions
Copy link

Latency summary

Current PR yields:

  • 🐢🐢 index QPS at 1001, delta to last 2 avg.: -37%
  • 🐎🐎🐎🐎 query QPS at 51, delta to last 2 avg.: +70%
  • 😶 avg flow time within 1.22 seconds, delta to last 2 avg.: -3%
  • 🐢🐢 import jina within 0.3007 seconds, delta to last 2 avg.: -11%

Breakdown

Version Index QPS Query QPS Avg Flow Time (s) Import Time (s)
current 1001 51 1.22 0.3007
2.6.4 1712 30 1.4736 0.3184
2.6.3 1491 29 1.0475 0.3622

Backed by latency-tracking. Further commits will update this comment.

Copy link
Member

@mapleeit mapleeit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python package system is so strange, from a NodeJSer's perspective. (node_modules is also a mess, tho )

Good hack. ; )

# as recommended by pip https://pip.pypa.io/en/latest/user_guide/#using-pip-from-your-program
import subprocess

subprocess.check_call(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesnt pip have a python package? This for instance would not work on my environment becauee I use pip3.7 for instance

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@numb3r3 could you throw some light here? i think you did something similar in hub.uses(from_source)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did this specifically because Pip says so: https://pip.pypa.io/en/latest/user_guide/#using-pip-from-your-program
Usually pip should be an alias to whatever pip version you are using, no?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same method is utilized in hub.uses(...). [sys.executorable, '-m', 'pip'] always use the executable pip associated with the current python version you are using.

@JoanFM JoanFM merged commit 1f2c863 into master Jan 24, 2022
@JoanFM JoanFM deleted the fix-docarray-upgrade branch January 24, 2022 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/setup This issue/PR affects setting up Jina size/S
Projects
None yet
Development

Successfully merging this pull request may close these issues.

module 'docarray' has no attribute '__version__' when upgrading jina from 2.6.4 to 2.6.5.dev60
5 participants