Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Release][Python] virtualenv not activated in verify-release-candidate.sh #38447

Closed
cyb70289 opened this issue Oct 25, 2023 · 6 comments · Fixed by #38486
Closed

[Release][Python] virtualenv not activated in verify-release-candidate.sh #38447

cyb70289 opened this issue Oct 25, 2023 · 6 comments · Fixed by #38486

Comments

@cyb70289
Copy link
Contributor

Describe the bug, including details regarding any error messages, version, and platform.

Tested on Arm64 ubuntu-20.04.
Try to verify 14.0.0 RC2 by TEST_DEFAULT=0 TEST_CPP=1 TEST_PYTHON=1 dev/release/verify-release-candidate.sh 14.0.0 2,
failed with below log:

===============================
Build and test Python libraries
===============================
└ Configuring Python 3 virtualenv...
└ Found interpreter Python 3.8.10: /usr/bin/python3
└ Using already created virtualenv at /tmp/arrow-14.0.0.bKwoL/venv-source
dev/release/verify-release-candidate.sh: line 541: /tmp/arrow-14.0.0.bKwoL/venv-source/bin/activate: No such file or directory
└ Installed pip packages cython>=0.29.31
Defaulting to user installation because normal site-packages is not writeable
Collecting cython>=0.29.31
  Downloading Cython-3.0.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (3.2 kB)
Requirement already satisfied: numpy in /home/cyb/.local/lib/python3.8/site-packages (1.23.1)
Collecting setuptools_scm<8.0.0
  Downloading setuptools_scm-7.1.0-py3-none-any.whl (43 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.8/43.8 kB 556.4 kB/s eta 0:00:00
Requirement already satisfied: setuptools in /home/cyb/.local/lib/python3.8/site-packages (68.2.2)
Requirement already satisfied: packaging>=20.0 in /usr/lib/python3/dist-packages (from setuptools_scm<8.0.0) (20.3)
Collecting typing-extensions (from setuptools_scm<8.0.0)
  Downloading typing_extensions-4.8.0-py3-none-any.whl.metadata (3.0 kB)
Requirement already satisfied: tomli>=1.0.0 in /home/cyb/.local/lib/python3.8/site-packages (from setuptools_scm<8.0.0) (2.0.1)
Downloading Cython-3.0.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.5/3.5 MB 9.9 MB/s eta 0:00:00
Downloading typing_extensions-4.8.0-py3-none-any.whl (31 kB)
DEPRECATION: distro-info 0.23ubuntu1 has a non-standard version number. pip 24.0 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of distro-info or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
DEPRECATION: gpg 1.13.1-unknown has a non-standard version number. pip 24.0 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of gpg or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
DEPRECATION: python-debian 0.1.36ubuntu1 has a non-standard version number. pip 24.0 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of python-debian or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
Installing collected packages: typing-extensions, cython, setuptools_scm
Successfully installed cython-3.0.4 setuptools_scm-7.1.0 typing-extensions-4.8.0
/tmp/arrow-14.0.0.bKwoL/apache-arrow-14.0.0/python /tmp/arrow-14.0.0.bKwoL/apache-arrow-14.0.0 ~/arrow
  File "setup.py", line 252
    print(f"-- Skipping build. Temp build {build_temp} does "
                                                            ^
SyntaxError: invalid syntax
Failed to verify release candidate. See /tmp/arrow-14.0.0.bKwoL for details.

The syntax error is probably because the default python version is python2 on my test machine.
The true error should be at the beginning of the log:
dev/release/verify-release-candidate.sh: line 541: /tmp/arrow-14.0.0.bKwoL/venv-source/bin/activate: No such file or directory

Related code is at https://github.com/apache/arrow/blob/main/dev/release/verify-release-candidate.sh#L531-L542
It checks existence of diretory /tmp/arrow-14.0.0.bKwoL/venv-source. If that dir exists, run bin/activate to activate virtualenv, otherwise create and activate a new virtualenv.
But on my test machine, that dir exists, but no bin/activate. So virtualenv is not activate. Still using python on the host.

ls /tmp/arrow-14.0.0.bKwoL/venv-source/bin
python  python3

Component(s)

Python, Release

@kou
Copy link
Member

kou commented Oct 25, 2023

Could you show logs around the "Creating python virtualenv at ..." message?

https://github.com/ursacomputing/crossbow/actions/runs/6613480564/job/17961403470#step:6:283 is the log in our nightly CI.

@cyb70289
Copy link
Contributor Author

=====================================
Build, install and test C++ libraries
=====================================
└ Configuring Python 3 virtualenv...
└ Found interpreter Python 3.8.10: /usr/bin/python3
└ Creating python virtualenv at /tmp/arrow-14.0.0.JMOGK/venv-source...
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt install python3.8-venv

You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.

Failing command: ['/tmp/arrow-14.0.0.JMOGK/venv-source/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']

dev/release/verify-release-candidate.sh: line 535: /tmp/arrow-14.0.0.JMOGK/venv-source/bin/activate: No such file or directory
Defaulting to user installation because normal site-packages is not writeable

I've installed python3.8-dev, looks it okay now.

@kou
Copy link
Member

kou commented Oct 25, 2023

Thanks.

It's strange that the verification script wasn't finished immediately when $python -m venv ${virtualenv} failed...

@kou
Copy link
Member

kou commented Oct 25, 2023

Hmm. set -e doesn't work in a function...:

$ cat /tmp/b.sh
set -e

a()
{
    false
    echo "a: failed after"
}
a || exit 1
echo "top level"
false
echo "failed after"
$ bash /tmp/b.sh
a: failed after
top level

@cyb70289
Copy link
Contributor Author

Looks it's also related to ||, without ||, error does triggers.

@kou
Copy link
Member

kou commented Oct 25, 2023

Wow! We may want to remove all || exit 1 in our verification script.

@kou kou closed this as completed in #38486 Dec 3, 2023
kou added a commit that referenced this issue Dec 3, 2023
### Rationale for this change

If we use "|| {exit,continue}", "set -x" doesn't work.

With "|| exit" ("false" in "a()" doesn't stop the shell execution):

```console
$ cat /tmp/with-or-exit.sh
set -e

a()
{
    false
    echo "a: failed after"
}
a || exit 1
echo "top level"
false
echo "failed after"
$ bash /tmp/with-or-exit.sh
a: failed after
top level
```

Without "|| exit" ("false" in "a()" stops the shell execution):

```console
$ cat /tmp/without-or-exit.sh
set -e

a()
{
    false
    echo "a: failed after"
}
a
echo "top level"
false
echo "failed after"
$ bash /tmp/without-or-exit.sh
```
### What changes are included in this PR?

* Remove needless `|| exit`
* Use `if` instead of `|| continue`

### Are these changes tested?

No.
* Closes: #38447

Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
@kou kou added this to the 15.0.0 milestone Dec 3, 2023
dgreiss pushed a commit to dgreiss/arrow that referenced this issue Feb 19, 2024
…38486)

### Rationale for this change

If we use "|| {exit,continue}", "set -x" doesn't work.

With "|| exit" ("false" in "a()" doesn't stop the shell execution):

```console
$ cat /tmp/with-or-exit.sh
set -e

a()
{
    false
    echo "a: failed after"
}
a || exit 1
echo "top level"
false
echo "failed after"
$ bash /tmp/with-or-exit.sh
a: failed after
top level
```

Without "|| exit" ("false" in "a()" stops the shell execution):

```console
$ cat /tmp/without-or-exit.sh
set -e

a()
{
    false
    echo "a: failed after"
}
a
echo "top level"
false
echo "failed after"
$ bash /tmp/without-or-exit.sh
```
### What changes are included in this PR?

* Remove needless `|| exit`
* Use `if` instead of `|| continue`

### Are these changes tested?

No.
* Closes: apache#38447

Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants