Use python 3.x task failing on linux private agent

[jrm346]

Required Information

Bug
Task: Use python, python script

Environment

• Server - Azure Pipelines
• Agent - Private (linux, latest Agent):

Issue Description

So i have been trying to get a private agent to work with Azure pipelines for a few days now and it has been a real mess. It was straight forward enough to install the agent itself on an Azure hosted linux VM and connect it to devops so that i could send jobs to it. The problems come from trying to install python on the agent.

First, there are almost no instructions whatsoever on how to install tools on agents. After reading though a lot of code on git hub I realized that i had to install python into a very specific directory on the agent. for example with python 3.6.8 i had to install to /agent/_work/_tool/Python/3.6.8/x64. and by install i mean compile from binaries to that location using ./configure, make, and make install. After doing that i have a copy of python which the 'Use python' task discovers. Unfortunately, simply installing from binaries has issues finding the ssl and results in the following error:

import _ssl             # if we can't import it, let the error propagate
ModuleNotFoundError: No module named '_ssl'

I'm still not sure how to fix this but am working on it. any help would be greatly appreciated.

Second, after installing python in the manner described above the 'Use python' task succeeds and prepends the right path to the PATH. Unfortunately, the 'python script' task uses the command which python to find which version of python to run. Installing python from binaries installs the program python3 so the running of a python script uses the system version of python, not the version named in the 'Use python' task. i can fix this by renaming python3 (and pip3) to python (and pip), but this seems somewhat hacky and i think should be fixed in the task its self.

Anyway i really think we need more documentation around how install tools on private agents and/or an installer tool for commonly used tools like python.

So if someone can provide me with detailed instructions as to how to install python on a private hosted agent so that the 'use python' task version of python is the one used and i can pip install and everything else that would be great.

Thanks.

[brcrista]

Setup

Did you see https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/tool/use-python-version?view=azure-devops#how-can-i-configure-a-self-hosted-agent-to-use-this-task ?

Yes, when we install Python on the Microsoft-hosted images, we follow the symlinking structure from PEP 394, and the Python Script task expects that too. But I will update the above doc to call that out explicitly.

SSL module

As for the ModuleNotFoundError: Python's ssl module is essentially a wrapper around a precompiled (that is, C code) shared object file. So that's saying that it can't find the shared object file for the module (which is called _ssl in the Python code).

I believe ssl is an "optional" module when you build from sources, meaning that if the build scripts don't find libssl to link against on the machine, it won't build it.

I'm not sure what distro you are on, but here's the packages to install from APT when building on Debian / Ubuntu:

apt-get install -y \
    build-essential \
    libbz2-dev \
    libdb-dev \
    libffi-dev \
    libgdbm-dev \
    liblzma-dev \
    libncursesw5-dev \
    libreadline-dev \
    libsqlite3-dev \
    libssl-dev \
    libz-dev \
    tk-dev \
    zlib1g-dev

[jrm346]

brcrista,

Thanks for getting back to me and i will give the stuff on the ssl module a go.

As for the setup i did go to where you put a link (which leads to the following git hub page https://github.com/microsoft/azure-pipelines-tool-lib/blob/master/docs/overview.md#tool-cache). That was how I knew what folder structure to install in. At the end of that page it mentions azure-pipelines-tool-lib which i assume is a tool for installing tools on agents. There seemed to be no instructions on how to use this and it pretty much just links to source code. I was really hoping there wound be a script packaged with agents that would allow for something like the following:

./tool-installer.sh python3.6

Thus making it easy to install the same tools that exist on microsoft hosted agents.

Anyway I'm not sure I understand what you are doing with symlinking. Are you creating a link in the tool folder (e.g. under x64/bin/) between python and python3 for every install of python (and presumably pip) so that they all respond to which python when the directory has been prepended to the path?

[brcrista]

Oh, that part about https://github.com/microsoft/azure-pipelines-tool-lib is directed to task implementers. That is a JavaScript library for working with this "tool cache." So in this case, if you lay out Python in this format, Use Python Version will be able to find it.

We actually do open-source our image generation scripts. Have a look at this part here: https://github.com/microsoft/azure-pipelines-image-generation/blob/179731fa11c0beda26b4a1e192710efb6850c207/images/linux/scripts/installers/hosted-tool-cache.sh#L11

The symlink structure is like python3.7 <- python3 <- python. So python3.7 is the actual interpreter, python3 is a symlink, and python is a symlink to the symlink.

[leantk]

This issue is stale because it has been open for a year with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days