Requirements setup for fprime in a Docker container, 'Matching python package version/distribution not found' family of errors, and related fun stuff #1608
Replies: 2 comments 2 replies
-
Perhaps you are missing the
I also have a devcontainer repo: https://github.com/astroesteban/fprime-devcontainer I use that regularly to develop for fprime inside of a Docker container with Visual Studio Code. You could probably take some of the stuff I have in there for your own purposes. |
Beta Was this translation helpful? Give feedback.
-
Before diving into the solution, it's worth noting that manual installation (without pip) of the error-invoking packages is not a good idea since the specific versions even when installed directly from PyPI for the correct version might interface with python or pip later on and cause issues (as one or both of them are redundant, and that's the root of this problem), apart from the fact that it can be cumbersome. The first step in itself while going that route would be to distinguish between what is there vs what's not, for which one could compare the currently installed pip packages with the ones from the requirements: python3 -m pip list > installedPackages.txt
diff installedPackages.txt <fprime-checkout-path>/requirements.txt Pretty sure there are better approaches to do the same, but I suspect it'll still be a bit of a pain. SolutionAs previously mentioned in this discussion and as documented elsewhere on the internet to be the core solution to this class of errors, one would have to uninstall both python(3) and pip(3), apart from completely removing the python packages that are partially installed or corrupted, and then install fresh versions of them using the correct method(s). Unfortunately, there are a handful of ways to go about this, and picking the right one(s) might not be obvious at times. (at least for me it wasn't at the beginning, with my inexperience of dabbling with python packages) By trial and error, a rule of thumb I learned and followed for the successful build was to use the correct package manager for what it's primarily intended to do, to not stick with one for all intents and purposes, and to definitely not overlap them. For e.g., if I were to introduce software such as python and pip, I'd use apt-get in Linux, and if I were to introduce python packages, I'd use pip. Mixing them (such as installing/updating python packages with both apt-get and pip) can lead to potential errors and misconfiguration. For uninstalling, one could use these: (I used the second option for my current build for clearing the leftover config files, but the first might be enough too, depending upon how python-based stuff is set up in your container) sudo apt-get remove python-pip python3-pip python python3
# assume the arguments above^ to be an ellipsis (...) henceforth
# better to use purge to remove the remnant config files:
sudo apt-get purge ...
# and possibly even better with this combination, if all dependencies need to be removed:
sudo apt-get --purge autoremove ... The next step is to get them back afresh (at some point, I'm sure I used sudo apt-get install python3.8 While this does set up python3.8 right, the old version (3.6.9 in my case) was still the one being defaulted to when As such, one would need to set sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 1 update-alternatives: using /usr/bin/python3.8 to provide /usr/bin/python3 (python3) in auto mode Given that I set the priority to the topmost integer (in descending order, like 3.6.9 at 2 would be of lower priority in comparison), it defaulted to that version, and running There is only one alternative in link group python3 (providing /usr/bin/python3): /usr/bin/python3.8
Nothing to configure. With Python3.8 set up, pip3 (or rather I guess I should start calling it For Python2.x and pip, I can't trace back and see my log as I did it in a separate CLI session (using Docker Desktop) prior to the current one that I've been running (and I didn't make notes of it), but I think I used these commands: (pulled this up from a thread that I was looking up around that time) sudo apt-get update && sudo apt-get -y install python python-pip
curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
python get-pip.py Either way, getting python3 and its corresponding pip package installer should be good enough for our case here. At the end of all this initial setup, I check their versions and the places where the executable binaries for them reside: (should be within recognized paths) declare -a pystuff=("python" "pip" "python3" "pip3")
for i in "${pystuff[@]}"; do "$i" --version; which "$i"; done Here's what they looked like for me prior to a successful pip install of the requirements, and at present: Python 2.7.17
/usr/bin/python
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)
/usr/bin/pip
Python 3.8.0
/usr/bin/python3
pip 22.2.1 from /home/user/.local/lib/python3.8/site-packages/pip (python 3.8)
/usr/local/bin/pip3 I proceeded to install the requirements and they finally worked for all the packages without a problem! sudo python3 -m pip install -r requirements.txt Note that using Again, feel free to exercise Speaking of cache also reminds me that redundant pip data can also lead to problems, such as during the deserialization of cached package entries: Collecting aniso8601==9.0.1 (from -r fprime/requirements.txt (line 1))
Cache entry deserialization failed, entry ignored
Using cached https://files.pythonhosted.org/packages/e3/04/e97c12dc034791d7b504860acfcdd2963fa21ae61eaca1c9d31245f812c3/aniso8601-9.0.1-py2.py3-none-any.whl
Collecting argcomplete==1.12.3 (from -r fprime/requirements.txt (line 2))
Cache entry deserialization failed, entry ignored
Using cached https://files.pythonhosted.org/packages/b7/9e/9dc74d330c07866d72f62d553fe8bdbe32786ff247a14e68b5659963e6bd/argcomplete-1.12.3-py2.py3-none-any.whl
Collecting arrow==1.2.2 (from -r fprime/requirements.txt (line 3))
Cache entry deserialization failed, entry ignored
... Given that it's an issue with the pip cache entirely, clearing or removing it ( Now my goal with this container was to experiment with the files that contained STest-based unit tests wrapped under GoogleTest macros, so as to run them using I think it might be helpful to share the steps for future reference if anyone else is trying to downgrade/update CMake (I see one discussion with relevancy), so here's what I would suggest based on what I did:
arch && cat /etc/os-release | head -2 x86_64
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)" Once you figured the one relevant for your arch and OS, download it to your computer (you could probably download it inside the Docker container through an URL directly, but I was lazy enough to avoid that route and instead just use the browser in my host machine to download the script, with the intention of copying it over to my container afterward). cmscript=cmake-3.24.0-rc5-linux-x86_64.sh # identify your version and replace accordingly!
sudo docker cp $cmscript <containerID>:/usr/share
sudo mv /usr/share/$cmscript /<directory>/
# for e.g., <directory> can be usr, usr/local, opt, etc.
sudo chmod +x /<directory>/$cmscript
sudo bash /<directory>/$cmscript --skip-license --exclude-subdir --prefix=/usr/local
sudo ln -s /usr/local/bin/cmake <alternatePath>
Optional, but feel free to do a final version and executable location check for all the 1requirements at this point, just to reassure oneselfdeclare -a allreqs=("cmake" "git" "gcc" "${pystuff[@]}")
for i in "${allreqs[@]}"; do "$i" --version; which "$i"; done If a container running OS X is to be used (I couldn't due to my tool's limited cross-platform support), I suspect this wouldn't be this complex with Homebrew. Just use |
Beta Was this translation helpful? Give feedback.
-
In the process of trying to set up fprime in a docker container (to facilitate easy execution of a tool that is configured therein, which in turn requires me to install this FSW framework atop if I were to integrate both) running Linux, I got the 'Could not find a version that satisfies the requirement
package_name
==semantic_version
' or 'No matching distribution found forxyz_package
==major.minor.patch
' type of errors when installing the required python packages via pip:If I were to remove
fprime-fpp
from the requirements file (with the intention of manually installing it later) and retry, I halt at the very next (lexicographically) required package:ERROR: Could not find a version that satisfies the requirement fprime-gds==3.1.1 (from versions: 1.5.5a1, 2.0, 3.0.0, 3.0.1, 3.0.2) ERROR: No matching distribution found for fprime-gds==3.1.1
Repeating what I did above once again, I'm presented with the next one:
ERROR: Could not find a version that satisfies the requirement fprime-tools==3.1.0 (from versions: 1.5.4.dev0, 1.5.4.dev1, 1.5.5a1, 2.0.0, 2.0.1, 2.0.2, 3.0.0, 3.0.1) ERROR: No matching distribution found for fprime-tools==3.1.0
And this goes on, although it's not that all packages continually throw this, and some down the line do, in fact, get installed (but it seems like the 'fprime' prefixed ones are among the chosen ones).
This PR shines some light on this, but the solution itself doesn't inherently work for all cases, as things can get more complicated if Python, pip, and some interlinked dependencies are not configured well. (probably worth mentioning that I haven't faced any problems outside the container while following the regular setup)
Beta Was this translation helpful? Give feedback.
All reactions