[ATH-1036] Fix ath-container and docker based tests #1037

raul-arabaolaza · 2023-02-22T17:04:11Z

See #1036 since #878 docker based tests are not run on ci.jenkins.io because docker can not be found on the ath image.

I have updated the docker installation process according to https://docs.docker.com/engine/install/ubuntu/ and also reverted c54ef5f#diff-59086a3e54e38095246aa3a051f9020274dbf005b4652a7808cd4c6c239c536eL67 so ath-user can see and run the docker command.

I have checked that the image builds and you can run docker inside as ath-user. Note that by running again docker tests we may find failures that were ignored before by accident. The purpose of this PR is not to solve those tests but to make the failures visible again. To check this is working we can compare the number of tests run for the WorkflowPluginTest suite in master and for this PR

Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
Ensure that the pull request title represents the desired changelog entry
Please describe what you did
Link to relevant issues in GitHub or Jira
Link to relevant pull requests, esp. upstream and downstream changes

Reverts jenkinsci@c54ef5f#diff-59086a3e54e38095246aa3a051f9020274dbf005b4652a7808cd4c6c239c536eL67

dduportal

LGTM from the jenkins-infra point of view

src/main/resources/ath-container/Dockerfile

raul-arabaolaza · 2023-02-23T08:30:32Z

Too many failures here, and I can not make sense of the error, I suspect it is because of an infra problem.

On the good side it looks like more tests are run, I will retrigger this build by using an empty commit

src/main/resources/ath-container/Dockerfile

jtnord · 2023-02-23T11:55:36Z

it looks like something has seriously regressed in either the LTS or was merged that broke pretty much all the tests.

https://ci.jenkins.io/job/Core/job/acceptance-test-harness/job/master/683/testReport/

could not see any merges around this timeline and is mostly around save - given the nature of the first few tests I looked at...

something is clearly wrong though.

https://ci.jenkins.io/job/Core/job/acceptance-test-harness/job/master/683/testReport/junit/core/FreestyleJobTest/java_11_jenkins_lts_split4___discardBuilds/ if you look at the last screenshot you see a jenkins version that is not an LTS version.

Could well be a bug in the junit attachment plugin but still....

Co-authored-by: James Nord <jtnord@users.noreply.github.com>

raul-arabaolaza · 2023-02-24T10:11:16Z

So, 56 skipped tests in this pr vs 226 in master -> 170 new tests run

Also for example https://ci.jenkins.io/job/Core/job/acceptance-test-harness/job/PR-1037/4/testReport/plugins/WorkflowPluginTest/java_11_jenkins_latest_split6___hello_world_from_git/ is run in this pr, but in https://ci.jenkins.io/job/Core/job/acceptance-test-harness/job/master/684/testReport/plugins/WorkflowPluginTest/java_11_jenkins_lts_split8___hello_world_from_git/ is skipped because docker is not found.

I do believe we can merge this, as failures are unrelated and the goal of unlocking docker tests is achieved

timja

I do believe we can merge this, as failures are unrelated and the goal of unlocking docker tests is achieved

Please do not merge this, a broken main build is the most important activity. Without a green baseline we cannot correctly evaluate this change and make sure there's no regression introduced, e.g. if this breaks any docker tests etc.

timja · 2023-02-27T09:04:50Z

it looks like something has seriously regressed in either the LTS or was merged that broke pretty much all the tests.

Filed #1039 which should be fixed before merging this.
From an initial look so far seem to be infra / container related, on my local machine the test I ran passed just fine.

raul-arabaolaza · 2023-02-27T09:05:24Z

@timja

What makes you believe those now ignored docker tests will be green? Actually in proprietary runs using another infra most of those tests are failing because of changes already merged in master. I can reproduce also locally without any infra changes involved.

I understand a green master would be desirable before merging any PR. But in this case I still believe it does not matter. Why? Because that green master will be an illusion. Even if all tests come back to be green in master that would be no guarantee at all as 170 docker based tests are skipped by error. Not because of this PR but other changes already merged. Should we continue ignoring those failures trying to get a fake green on master? I believe is better to get all tests run again and then confidently get master back to a real green.

Or in other words, I do prefer a legit red build that I can fix with confidence step by step than trying to fix something that will be anyway broken without me knowing. Because right now any fix that I may propose to for example WorkflowPluginTest#hello_world_from_git will not be validated until this pr is merged. And I will not create a gigantic PR that is gonna restore docker based tests and contain unrelated ( to the original intention of the PR) test fixes. I would rather do those in separate, small, isolated and easily reviewable PRs after the PR builder is working.

timja · 2023-02-27T09:15:46Z

Or in other words, I do prefer a legit red build that I can fix with confidence step by step than trying to fix something that will be anyway broken without me knowing. Because right now any fix that I may propose to for example WorkflowPluginTest#hello_world_from_git will not be validated until this pr is merged. And I will not create a gigantic PR that is gonna restore docker based tests and contain unrelated ( to the original intention of the PR) test fixes. I would rather do those in separate, small, isolated and easily reviewable PRs after the PR builder is working.

You (or anyone else) can branch from this PR to fix those as well. Merging on top of red is something I absolutely do not support.

(I've ran out of time to check them for now but the next thing I would try it reverting #1024 to see if they go back to green)

raul-arabaolaza · 2023-02-27T15:52:22Z

#1024 has proved blind faith in green builds is not a guarantee of success. But anyway I do not have more time to dedicate to this for the moment. If anyone wants to use this as base for future PRs please go forward.

timja · 2023-03-06T13:19:07Z

Seems lots of docker tests are failing for not being able to build the image currently

raul-arabaolaza · 2023-03-06T15:01:17Z

Let me check, I had problems with old docker versions locally

raul-arabaolaza · 2023-03-06T15:04:52Z

Not all tests are failing because of that, of my random 4 picks it was 50%, anyway, let's see what is failing

raul-arabaolaza · 2023-03-06T15:06:04Z

Oh, 2.977 Package viewvc is not available, but is referred to by another package. #6 2.977 This may mean that the package is missing, has been obsoleted, or #6 2.977 is only available from another source #6 2.977 #6 3.040 E: Package 'viewvc' has no installation candidate

Those kind of errors should be easy to fix or to die trying

raul-arabaolaza · 2023-03-06T15:19:14Z

viewvc/viewvc#310

See refenrence in Dockerfile

raul-arabaolaza · 2023-03-06T16:02:00Z

https://community.developer.atlassian.com/t/the-repository-https-packages-atlassian-com-atlassian-sdk-deb-stable-release-is-not-signed/36901/14

raul-arabaolaza · 2023-03-06T16:57:17Z

So, subversion and (I hope) jira plugin should be fixed on the last run, next step git plugin tests

raul-arabaolaza · 2023-03-07T11:03:18Z

DeplyPluginTests should be fixed now

As tomcat 7 is not java11 compatible

jtnord · 2023-03-07T16:48:15Z

src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/GitContainer/Dockerfile

@@ -3,7 +3,7 @@
 # and prepares for execution of gitplugin tests.
 #

-FROM ubuntu:22.04
+FROM ubuntu:xenial


most likely the line 24 and 19 are incorrect

RUN cat /home/git/unsafe.pub >> /home/git/.ssh/authorized_keys
is likely creating the file using the default keymask (infact the line 19 mkdir -p /home/git/.ssh is probably just as bad.

the .ssh folder should be 0700 and be owned by the git user (not root)

line 30 RUN chown -R git /home/git changes the ownership back but would not address any permission changes.

I have had some success with this suggestion locally, let's see what the pr builder says

+1 with @jtnord . I'm not sure how this Java code works to spin up Docker container (and I'm too scared to try it), but I got the following updated Dockerfile here tested as below (as a first step) and passed through hadolint (which has very very good tips and actionnables, like shellcheck):

FROM ubuntu:22.04 SHELL ["/bin/bash", "-o", "pipefail", "-c"] # install SSHD, Git and zip, always on the latest available version # hadolint ignore=DL3008 RUN apt-get update \ && apt-get install --yes --no-install-recommends \ ca-certificates \ openssh-server \ git \ zip \ && rm -rf /var/lib/apt/lists/* # create a git user and setup its .ssh dir (including public key) COPY unsafe.pub /home/git/.ssh/authorized_keys RUN useradd git -d /home/git \ && echo "git:git" | chpasswd \ && mkdir -p /var/run/sshd \ && chmod 0700 /home/git/.ssh \ && chmod 0600 /home/git/.ssh/* \ && chown -R git /home/git # run SSHD in the foreground with error messages to stderr ENTRYPOINT ["/usr/sbin/sshd"] CMD ["-D","-e"]

$ pwd <my workspace>/acceptance-test-harness/src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/GitContainer $ docker build -t sshd ./ # Success $ docker run --rm --detach --name=sshd-test --publish 2222:22 sshd # Run in background the container $ ssh -i ./unsafe -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -p2222 git@localhost whoami # .. git ## It works

raul-arabaolaza · 2023-03-07T20:21:01Z

@timja @dduportal @jtnord Can you re review please? AFAICT the JobConfigHistoryTest is a flake, it has been working and failing randomly during the course of this PR while my changes should not affect it at all

raul-arabaolaza · 2023-03-07T20:24:02Z

src/main/resources/org/jenkinsci/test/acceptance/docker/fixtures/GitContainer/Dockerfile

+RUN chmod 700 /home/git/.ssh
+RUN chmod 600 /home/git/.ssh/authorized_keys
+
+RUN echo "HostbasedAcceptedKeyTypes +ssh-rsa\nHostKeyAlgorithms +ssh-rsa\nPubkeyAcceptedKeyTypes +ssh-rsa" >> /etc/ssh/sshd_config


We need this for jsch to be able to sftp into the container, I tried with more modern algorithms for the keys but jsch did not supported them

raul-arabaolaza · 2023-03-07T20:24:59Z

src/test/java/plugins/DeployPluginTest.java


    @Test
    @Native("bash")
    @WithCredentials(credentialType = WithCredentials.USERNAME_PASSWORD, values = {"admin", "tomcat"}, id = "tomcat")
-    public void deploy_sample_webapp_to_tomcat7() throws IOException, InterruptedException {
+    public void deploy_sample_webapp_to_tomcat10() throws IOException, InterruptedException {


I do not believe deploying in tomcat 10 vs deploying in tomcat 7 changes the intent of the test

timja · 2023-03-07T21:04:39Z

@timja @dduportal @jtnord Can you re review please? AFAICT the JobConfigHistoryTest is a flake, it has been working and failing randomly during the course of this PR while my changes should not affect it at all

You could file an issue and disable it if you don’t have time to look at it.

raul-arabaolaza · 2023-03-08T08:27:03Z

Issues created, see link, and test ignored

raul-arabaolaza · 2023-03-08T08:27:34Z

@timja Can you please reenable auto merge or merge if green?

dduportal

👏 🚀

timja · 2023-08-11T22:05:57Z

Caused #1322

[ATH-1036] Fix docker image so docker is available

cb0559c

Reverts jenkinsci@c54ef5f#diff-59086a3e54e38095246aa3a051f9020274dbf005b4652a7808cd4c6c239c536eL67

dduportal approved these changes Feb 22, 2023

View reviewed changes

jtnord reviewed Feb 22, 2023

View reviewed changes

src/main/resources/ath-container/Dockerfile Show resolved Hide resolved

Empty-Commit to retrigger

1dadf03

jtnord approved these changes Feb 23, 2023

View reviewed changes

src/main/resources/ath-container/Dockerfile Outdated Show resolved Hide resolved

Update src/main/resources/ath-container/Dockerfile

2e0e03f

Co-authored-by: James Nord <jtnord@users.noreply.github.com>

timja requested changes Feb 24, 2023

View reviewed changes

raul-arabaolaza mentioned this pull request Feb 28, 2023

Revert "Bump leftover docker images to ubuntu 22.04" #1040

Closed

Merge branch 'master' into ATH-1036

e6f8b01

[ATH-1036] Use oldstable to get viewvc

39418d0

See refenrence in Dockerfile

[ATH-1036] Fix atlassian-sdk installation

12605fc

raul-arabaolaza added 3 commits March 7, 2023 11:09

Partially revert jenkinsci#1040

8f2d3a4

[ATH-1036] Make sure the container contains the ssh client

77c147c

[ATH-1036] Partially revert jenkinsci#1048

fc18db5

Empty-Commit to retrigger

a87ee0e

[ATH-1036] Use tomcat 10 with ubuntu 22.04 for DeployPluginTest

a03f0c0

As tomcat 7 is not java11 compatible

jtnord reviewed Mar 7, 2023

View reviewed changes

raul-arabaolaza added 2 commits March 7, 2023 19:12

[ATH-1036] Try to use ubuntu 22.04

5bcae05

Empty-Commit to retrigger

36555b3

raul-arabaolaza requested review from timja and dduportal and removed request for timja and dduportal March 7, 2023 20:19

raul-arabaolaza commented Mar 7, 2023

View reviewed changes

timja approved these changes Mar 7, 2023

View reviewed changes

timja enabled auto-merge (squash) March 7, 2023 21:05

timja added the bug label Mar 7, 2023

raul-arabaolaza mentioned this pull request Mar 8, 2023

JobConfigHistoryPluginTest is flaky #1052

Open

Ignore flaky test until issue 1052 is resolved

fcabccf

auto-merge was automatically disabled March 8, 2023 08:26
Head branch was pushed to by a user without write access

raul-arabaolaza changed the title ~~[ATH-1036] Fix docker image so docker based tests are run~~ [ATH-1036] Fix ath-container and docker based tests Mar 8, 2023

Empty-Commit to retrigger

6a2c26e

NotMyFault approved these changes Mar 8, 2023

View reviewed changes

dduportal approved these changes Mar 8, 2023

View reviewed changes

dduportal merged commit 8ac1ea0 into jenkinsci:master Mar 8, 2023

raul-arabaolaza deleted the ATH-1036 branch March 8, 2023 12:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ATH-1036] Fix ath-container and docker based tests #1037

[ATH-1036] Fix ath-container and docker based tests #1037

raul-arabaolaza commented Feb 22, 2023

dduportal left a comment

raul-arabaolaza commented Feb 23, 2023

jtnord commented Feb 23, 2023 •

edited

Loading

raul-arabaolaza commented Feb 24, 2023

timja left a comment

timja commented Feb 27, 2023

raul-arabaolaza commented Feb 27, 2023

timja commented Feb 27, 2023

raul-arabaolaza commented Feb 27, 2023 •

edited

Loading

timja commented Mar 6, 2023

raul-arabaolaza commented Mar 6, 2023

raul-arabaolaza commented Mar 6, 2023

raul-arabaolaza commented Mar 6, 2023

raul-arabaolaza commented Mar 6, 2023

raul-arabaolaza commented Mar 6, 2023

raul-arabaolaza commented Mar 6, 2023

raul-arabaolaza commented Mar 7, 2023

jtnord Mar 7, 2023 •

edited

Loading

raul-arabaolaza Mar 7, 2023

dduportal Mar 7, 2023

raul-arabaolaza commented Mar 7, 2023

raul-arabaolaza Mar 7, 2023

raul-arabaolaza Mar 7, 2023

timja commented Mar 7, 2023

raul-arabaolaza commented Mar 8, 2023

raul-arabaolaza commented Mar 8, 2023 •

edited

Loading

dduportal left a comment

timja commented Aug 11, 2023

[ATH-1036] Fix ath-container and docker based tests #1037

[ATH-1036] Fix ath-container and docker based tests #1037

Conversation

raul-arabaolaza commented Feb 22, 2023

dduportal left a comment

Choose a reason for hiding this comment

raul-arabaolaza commented Feb 23, 2023

jtnord commented Feb 23, 2023 • edited Loading

raul-arabaolaza commented Feb 24, 2023

timja left a comment

Choose a reason for hiding this comment

timja commented Feb 27, 2023

raul-arabaolaza commented Feb 27, 2023

timja commented Feb 27, 2023

raul-arabaolaza commented Feb 27, 2023 • edited Loading

timja commented Mar 6, 2023

raul-arabaolaza commented Mar 6, 2023

raul-arabaolaza commented Mar 6, 2023

raul-arabaolaza commented Mar 6, 2023

raul-arabaolaza commented Mar 6, 2023

raul-arabaolaza commented Mar 6, 2023

raul-arabaolaza commented Mar 6, 2023

raul-arabaolaza commented Mar 7, 2023

jtnord Mar 7, 2023 • edited Loading

Choose a reason for hiding this comment

raul-arabaolaza Mar 7, 2023

Choose a reason for hiding this comment

dduportal Mar 7, 2023

Choose a reason for hiding this comment

raul-arabaolaza commented Mar 7, 2023

raul-arabaolaza Mar 7, 2023

Choose a reason for hiding this comment

raul-arabaolaza Mar 7, 2023

Choose a reason for hiding this comment

timja commented Mar 7, 2023

raul-arabaolaza commented Mar 8, 2023

raul-arabaolaza commented Mar 8, 2023 • edited Loading

dduportal left a comment

Choose a reason for hiding this comment

timja commented Aug 11, 2023

jtnord commented Feb 23, 2023 •

edited

Loading

raul-arabaolaza commented Feb 27, 2023 •

edited

Loading

jtnord Mar 7, 2023 •

edited

Loading

raul-arabaolaza commented Mar 8, 2023 •

edited

Loading