Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix fasttest by pinning pip dependencies #59256

Merged
merged 1 commit into from Jan 26, 2024

Conversation

azat
Copy link
Collaborator

@azat azat commented Jan 26, 2024

After image update it prints the following warning:

/ClickHouse/tests/queries/0_stateless/01558_ttest_scipy.python:5: DeprecationWarning:
Pyarrow will become a required dependency of pandas in the next major release of pandas (pandas 3.0),
(to allow more performant data types, such as the Arrow string type, and better interoperability with other libraries)
but was not found to be installed on your system.
If this would cause problems for you,
please provide us feedback at https://github.com/pandas-dev/pandas/issues/54466

Pyarrow can be added as well, but according to the comment in the issue it bloats the image size 170%, so let's simply downgrade it for now and see how that issue will be resolved (or the warning could be suppressed of course).

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

@robot-clickhouse
Copy link
Member

robot-clickhouse commented Jan 26, 2024

This is an automated comment for commit 92d96c4 with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Successful checks
Check nameDescriptionStatus
ClickBenchRuns [ClickBench](https://github.com/ClickHouse/ClickBench/) with instant-attach table✅ success
ClickHouse build checkBuilds ClickHouse in various configurations for use in further steps. You have to fix the builds that fail. Build logs often has enough information to fix the error, but you might have to reproduce the failure locally. The cmake options can be found in the build log, grepping for cmake. Use these options and follow the general build process✅ success
Compatibility checkChecks that clickhouse binary runs on distributions with old libc versions. If it fails, ask a maintainer for help✅ success
Docker server and keeper imagesThere's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS✅ success
Docs checkThere's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS✅ success
Fast testsThere's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS✅ success
Flaky testsChecks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integrational tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc✅ success
Install packagesChecks that the built packages are installable in a clear environment✅ success
Mergeable CheckChecks if all other necessary checks are successful✅ success
Push to DockerhubThe check for building and pushing the CI related docker images to docker hub✅ success
SqllogicRun clickhouse on the sqllogic test set against sqlite and checks that all statements are passed✅ success
Stateful testsRuns stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc✅ success
Style checkThere's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS✅ success
Unit testsRuns the unit tests for different release types✅ success
Check nameDescriptionStatus
CI runningA meta-check that indicates the running CI. Normally, it's in success or pending state. The failed status indicates some problems with the PR⏳ pending
Stateless testsRuns stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc❌ error

@Algunenano Algunenano self-assigned this Jan 26, 2024
@Algunenano
Copy link
Member

I'm also looking in parallel into updating all of the dependencies (since newer Python requires newer deps) with:

RUN pip3 install numpy==1.26.3 scipy==1.12.0 pandas==2.2.0 pyarrow==15.0.0 Jinja2==3.1.3

Waiting on CI to see what works

@azat
Copy link
Collaborator Author

azat commented Jan 26, 2024

I'm also looking in parallel into updating all of the dependencies (since newer Python requires newer deps) with:

Great, I also though about pinning everything! Is the PR in some private repo? Or there is no PR yet?

@Algunenano
Copy link
Member

Is the PR in some private repo?

Yes, but I'll bring it here if the CI is ok.

@azat
Copy link
Collaborator Author

azat commented Jan 26, 2024

pyarrow==15.0.0

As for adding pyarrow, it will increase image size to 100MB, and this is the thing that I was trying to avoid (maybe pandas will eventually remove requirement for pyarrow, since it seems that some people are unhappy)

@Algunenano
Copy link
Member

You are right, I didn't check. Let's try pip install numpy==1.26.3 scipy==1.12.0 pandas==1.5.3 Jinja2==3.1.3 here if you don't mind

After image update it prints the following warning:

    /ClickHouse/tests/queries/0_stateless/01558_ttest_scipy.python:5: DeprecationWarning:
    Pyarrow will become a required dependency of pandas in the next major release of pandas (pandas 3.0),
    (to allow more performant data types, such as the Arrow string type, and better interoperability with other libraries)
    but was not found to be installed on your system.
    If this would cause problems for you,
    please provide us feedback at pandas-dev/pandas#54466

Pyarrow can be added as well, but according to the comment in the issue
it bloats the image size 170%, so let's simply downgrade it for now and
see how that issue will be resolved (or the warning could be suppressed
of course).

And pin other dependencies as well to avoid further failures.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
@azat
Copy link
Collaborator Author

azat commented Jan 26, 2024

Sure, I've adjusted the patch. And I've tested it locally image had been successfully built, let's see what CI will say.

@azat azat changed the title Fix fasttest by pinning pandas dependency Fix fasttest by pinning pip dependencies Jan 26, 2024
@azat
Copy link
Collaborator Author

azat commented Jan 26, 2024

Fast test passed

@Felixoid Felixoid merged commit 4286634 into ClickHouse:master Jan 26, 2024
84 of 169 checks passed
@azat azat deleted the tests/fasttest branch January 26, 2024 15:06
@Algunenano Algunenano added pr-must-backport Pull request should be backported intentionally. Use this label with great care! pr-must-backport-cloud labels Jan 26, 2024
robot-clickhouse-ci-1 added a commit that referenced this pull request Jan 26, 2024
…84f780e083dbdf797aaa2815467f5

Cherry pick #59256 to 23.3: Fix fasttest by pinning pip dependencies
robot-clickhouse-ci-1 added a commit that referenced this pull request Jan 26, 2024
…84f780e083dbdf797aaa2815467f5

Cherry pick #59256 to 23.8: Fix fasttest by pinning pip dependencies
robot-clickhouse-ci-1 added a commit that referenced this pull request Jan 26, 2024
…b84f780e083dbdf797aaa2815467f5

Cherry pick #59256 to 23.10: Fix fasttest by pinning pip dependencies
robot-clickhouse-ci-1 added a commit that referenced this pull request Jan 26, 2024
…b84f780e083dbdf797aaa2815467f5

Cherry pick #59256 to 23.11: Fix fasttest by pinning pip dependencies
robot-clickhouse-ci-1 added a commit that referenced this pull request Jan 26, 2024
…b84f780e083dbdf797aaa2815467f5

Cherry pick #59256 to 23.12: Fix fasttest by pinning pip dependencies
@robot-clickhouse-ci-1 robot-clickhouse-ci-1 added the pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore label Jan 26, 2024
robot-clickhouse-ci-1 added a commit that referenced this pull request Jan 26, 2024
Backport #59256 to 23.12: Fix fasttest by pinning pip dependencies
@Felixoid
Copy link
Member

@Algunenano, the backports are unnecessary, actually. Fast tests don't run for release branches anyhow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore pr-backports-created-cloud pr-must-backport Pull request should be backported intentionally. Use this label with great care! pr-must-backport-cloud pr-not-for-changelog This PR should not be mentioned in the changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants