Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[7.17](backport #35780) Upgrade to Go 1.19.10 and handle changes in golang Debian base image #35840

Merged
merged 17 commits into from
Jun 21, 2023

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Jun 20, 2023

This is an automatic backport of pull request #35780 done by Mergify.
Cherry-pick of 62374dd has failed:

On branch mergify/bp/7.17/pr-35780
Your branch is up to date with 'origin/7.17'.

You are currently cherry-picking commit 62374dd251.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   .go-version
	modified:   .golangci.yml
	modified:   CHANGELOG.next.asciidoc
	modified:   auditbeat/Dockerfile
	modified:   dev-tools/mage/target/integtest/integtest.go
	modified:   heartbeat/Dockerfile
	modified:   metricbeat/magefile.go
	modified:   metricbeat/module/aerospike/_meta/Dockerfile
	modified:   metricbeat/module/haproxy/_meta/Dockerfile
	modified:   metricbeat/module/kafka/_meta/Dockerfile
	modified:   metricbeat/module/mongodb/_meta/Dockerfile
	modified:   metricbeat/module/munin/_meta/Dockerfile
	modified:   metricbeat/module/rabbitmq/_meta/Dockerfile
	modified:   packetbeat/Dockerfile
	modified:   testing/environments/docker/kafka/Dockerfile
	modified:   x-pack/functionbeat/Dockerfile
	modified:   x-pack/metricbeat/magefile.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   dev-tools/mage/pytest.go
	both modified:   libbeat/docs/version.asciidoc
	both modified:   libbeat/tests/system/requirements.txt
	both modified:   libbeat/tests/system/requirements_aix.txt
	both modified:   metricbeat/Dockerfile

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally


Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

  • @Mergifyio refresh will re-evaluate the rules
  • @Mergifyio rebase will rebase this PR on its base branch
  • @Mergifyio update will merge the base branch into this PR
  • @Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

  • look at your merge queues
  • generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.com

v1v and others added 5 commits January 24, 2023 07:23
(cherry picked from commit a5f15b6)

# Conflicts:
#	.ci/beats-tester.groovy
#	.ci/build-docker-images.groovy
#	.ci/heartbeat-synthetics.groovy
#	x-pack/filebeat/Jenkinsfile.yml
#	x-pack/metricbeat/Jenkinsfile.yml
…35780)

* Explicitly install netcat-openbsd everywhere.

The plain netcat package appears to have been removed in the latest
golang:1.19.10 base image.

* Setup a virtualenv in Docker containers.

This fixes the PEP668 error about "externally managed environments"
which prevents modifying the system python installation.

* Revert "Revert "[Automation] Bump Golang version to 1.19.10 (#35751)" (#35784)"

This reverts commit 27b1799.

* Reword CHANGELOG.next.asciidoc

Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com>

* Improve PEP668 comments.

* Install latest PyYAML in test containers.

* Add check to see if already in a venv.

* Upgrade pytest and py.

Remove dependency on pytest-otel as it prevented upgrading pytest.

* Check for pre-set virtualenv earlier.

* Always create a fresh virtualenv in Docker.

Remove duplicate dependencies from the docker files.

* Fix metricbeat integration tests.

* Force install Python dependencies for Metricbeat.

---------

Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com>
(cherry picked from commit 62374dd)

# Conflicts:
#	dev-tools/mage/pytest.go
#	libbeat/docs/version.asciidoc
#	libbeat/tests/system/requirements.txt
#	libbeat/tests/system/requirements_aix.txt
#	metricbeat/Dockerfile
@mergify mergify bot requested review from a team as code owners June 20, 2023 19:30
@mergify mergify bot added the backport label Jun 20, 2023
@mergify mergify bot requested a review from a team as a code owner June 20, 2023 19:30
@mergify mergify bot added the conflicts There is a conflict in the backported pull request label Jun 20, 2023
@mergify mergify bot requested review from pierrehilbert and fearful-symmetry and removed request for a team June 20, 2023 19:30
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 20, 2023
@botelastic
Copy link

botelastic bot commented Jun 20, 2023

This pull request doesn't have a Team:<team> label.

@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 20, 2023

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-06-21T12:35:39.210+0000

  • Duration: 61 min 55 sec

Test stats 🧪

Test Results
Failed 4
Passed 23
Skipped 4
Total 31

Test errors 4

Expand to view the tests failures

test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0
    Expand to view the error details

     Failed: Timeout >90.0s 
    

    Expand to view the stacktrace

     self = <test_activemq.ActiveMqTest_0 testMethod=test_queue_metrics_collected>
    
        @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, 'integration test')
        def test_queue_metrics_collected(self):
    >       self.verify_destination_metrics_collection('queue')
    
    module/activemq/test_activemq.py:96: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    module/activemq/test_activemq.py:70: in verify_destination_metrics_collection
        conn.disconnect()
    /opt/venv/lib/python3.11/site-packages/stomp/connect.py:185: in disconnect
        self.transport.stop()
    /opt/venv/lib/python3.11/site-packages/stomp/transport.py:122: in stop
        self.__receiver_thread_exit_condition.wait()
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    
    self = <Condition(<unlocked _thread.RLock object owner=0 count=0 at 0x7f6f1d4eb980>, 0)>
    timeout = None
    
        def wait(self, timeout=None):
            """Wait until notified or until a timeout occurs.
        
            If the calling thread has not acquired the lock when this method is
            called, a RuntimeError is raised.
        
            This method releases the underlying lock, and then blocks until it is
            awakened by a notify() or notify_all() call for the same condition
            variable in another thread, or until the optional timeout occurs. Once
            awakened or timed out, it re-acquires the lock and returns.
        
            When the timeout argument is present and not None, it should be a
            floating point number specifying a timeout for the operation in seconds
            (or fractions thereof).
        
            When the underlying lock is an RLock, it is not released using its
            release() method, since this may not actually unlock the lock when it
            was acquired multiple times recursively. Instead, an internal interface
            of the RLock class is used, which really unlocks it even when it has
            been recursively acquired several times. Another internal interface is
            then used to restore the recursion level when the lock is reacquired.
        
            """
            if not self._is_owned():
                raise RuntimeError("cannot wait on un-acquired lock")
            waiter = _allocate_lock()
            waiter.acquire()
            self._waiters.append(waiter)
            saved_state = self._release_save()
            gotit = False
            try:    # restore state no matter what (e.g., KeyboardInterrupt)
                if timeout is None:
    >               waiter.acquire()
    E               Failed: Timeout >90.0s
    
    /usr/lib/python3.11/threading.py:320: Failed 
    

test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0
    Expand to view the error details

     Failed: Timeout >90.0s 
    

    Expand to view the stacktrace

     self = <test_activemq.ActiveMqTest_0 testMethod=test_topic_metrics_collected>
    
        @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, 'integration test')
        def test_topic_metrics_collected(self):
    >       self.verify_destination_metrics_collection('topic')
    
    module/activemq/test_activemq.py:100: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    module/activemq/test_activemq.py:70: in verify_destination_metrics_collection
        conn.disconnect()
    /opt/venv/lib/python3.11/site-packages/stomp/connect.py:185: in disconnect
        self.transport.stop()
    /opt/venv/lib/python3.11/site-packages/stomp/transport.py:122: in stop
        self.__receiver_thread_exit_condition.wait()
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    
    self = <Condition(<unlocked _thread.RLock object owner=0 count=0 at 0x7f6f1d11e200>, 0)>
    timeout = None
    
        def wait(self, timeout=None):
            """Wait until notified or until a timeout occurs.
        
            If the calling thread has not acquired the lock when this method is
            called, a RuntimeError is raised.
        
            This method releases the underlying lock, and then blocks until it is
            awakened by a notify() or notify_all() call for the same condition
            variable in another thread, or until the optional timeout occurs. Once
            awakened or timed out, it re-acquires the lock and returns.
        
            When the timeout argument is present and not None, it should be a
            floating point number specifying a timeout for the operation in seconds
            (or fractions thereof).
        
            When the underlying lock is an RLock, it is not released using its
            release() method, since this may not actually unlock the lock when it
            was acquired multiple times recursively. Instead, an internal interface
            of the RLock class is used, which really unlocks it even when it has
            been recursively acquired several times. Another internal interface is
            then used to restore the recursion level when the lock is reacquired.
        
            """
            if not self._is_owned():
                raise RuntimeError("cannot wait on un-acquired lock")
            waiter = _allocate_lock()
            waiter.acquire()
            self._waiters.append(waiter)
            saved_state = self._release_save()
            gotit = False
            try:    # restore state no matter what (e.g., KeyboardInterrupt)
                if timeout is None:
    >               waiter.acquire()
    E               Failed: Timeout >90.0s
    
    /usr/lib/python3.11/threading.py:320: Failed 
    

test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1
    Expand to view the error details

     Failed: Timeout >90.0s 
    

    Expand to view the stacktrace

     self = <test_activemq.ActiveMqTest_1 testMethod=test_queue_metrics_collected>
    
        @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, 'integration test')
        def test_queue_metrics_collected(self):
    >       self.verify_destination_metrics_collection('queue')
    
    module/activemq/test_activemq.py:96: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    module/activemq/test_activemq.py:70: in verify_destination_metrics_collection
        conn.disconnect()
    /opt/venv/lib/python3.11/site-packages/stomp/connect.py:185: in disconnect
        self.transport.stop()
    /opt/venv/lib/python3.11/site-packages/stomp/transport.py:122: in stop
        self.__receiver_thread_exit_condition.wait()
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    
    self = <Condition(<unlocked _thread.RLock object owner=0 count=0 at 0x7f6f1d513d00>, 0)>
    timeout = None
    
        def wait(self, timeout=None):
            """Wait until notified or until a timeout occurs.
        
            If the calling thread has not acquired the lock when this method is
            called, a RuntimeError is raised.
        
            This method releases the underlying lock, and then blocks until it is
            awakened by a notify() or notify_all() call for the same condition
            variable in another thread, or until the optional timeout occurs. Once
            awakened or timed out, it re-acquires the lock and returns.
        
            When the timeout argument is present and not None, it should be a
            floating point number specifying a timeout for the operation in seconds
            (or fractions thereof).
        
            When the underlying lock is an RLock, it is not released using its
            release() method, since this may not actually unlock the lock when it
            was acquired multiple times recursively. Instead, an internal interface
            of the RLock class is used, which really unlocks it even when it has
            been recursively acquired several times. Another internal interface is
            then used to restore the recursion level when the lock is reacquired.
        
            """
            if not self._is_owned():
                raise RuntimeError("cannot wait on un-acquired lock")
            waiter = _allocate_lock()
            waiter.acquire()
            self._waiters.append(waiter)
            saved_state = self._release_save()
            gotit = False
            try:    # restore state no matter what (e.g., KeyboardInterrupt)
                if timeout is None:
    >               waiter.acquire()
    E               Failed: Timeout >90.0s
    
    /usr/lib/python3.11/threading.py:320: Failed 
    

test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1
    Expand to view the error details

     Failed: Timeout >90.0s 
    

    Expand to view the stacktrace

     self = <test_activemq.ActiveMqTest_1 testMethod=test_topic_metrics_collected>
    
        @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, 'integration test')
        def test_topic_metrics_collected(self):
    >       self.verify_destination_metrics_collection('topic')
    
    module/activemq/test_activemq.py:100: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    module/activemq/test_activemq.py:70: in verify_destination_metrics_collection
        conn.disconnect()
    /opt/venv/lib/python3.11/site-packages/stomp/connect.py:185: in disconnect
        self.transport.stop()
    /opt/venv/lib/python3.11/site-packages/stomp/transport.py:122: in stop
        self.__receiver_thread_exit_condition.wait()
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    
    self = <Condition(<unlocked _thread.RLock object owner=0 count=0 at 0x7f6f1d47f540>, 0)>
    timeout = None
    
        def wait(self, timeout=None):
            """Wait until notified or until a timeout occurs.
        
            If the calling thread has not acquired the lock when this method is
            called, a RuntimeError is raised.
        
            This method releases the underlying lock, and then blocks until it is
            awakened by a notify() or notify_all() call for the same condition
            variable in another thread, or until the optional timeout occurs. Once
            awakened or timed out, it re-acquires the lock and returns.
        
            When the timeout argument is present and not None, it should be a
            floating point number specifying a timeout for the operation in seconds
            (or fractions thereof).
        
            When the underlying lock is an RLock, it is not released using its
            release() method, since this may not actually unlock the lock when it
            was acquired multiple times recursively. Instead, an internal interface
            of the RLock class is used, which really unlocks it even when it has
            been recursively acquired several times. Another internal interface is
            then used to restore the recursion level when the lock is reacquired.
        
            """
            if not self._is_owned():
                raise RuntimeError("cannot wait on un-acquired lock")
            waiter = _allocate_lock()
            waiter.acquire()
            self._waiters.append(waiter)
            saved_state = self._release_save()
            gotit = False
            try:    # restore state no matter what (e.g., KeyboardInterrupt)
                if timeout is None:
    >               waiter.acquire()
    E               Failed: Timeout >90.0s
    
    /usr/lib/python3.11/threading.py:320: Failed 
    

Steps errors 6

Expand to view the steps failures

Setup kind
  • Took 2 min 28 sec . View more details here
  • Description: .ci/scripts/kind-setup.sh
Integration tests
  • Took 3 min 2 sec . View more details here
  • Description: MODULE=kubernetes make -C metricbeat integration-tests
x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest
  • Took 20 min 1 sec . View more details here
  • Description: mage pythonIntegTest
x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest
  • Took 14 min 44 sec . View more details here
  • Description: mage pythonIntegTest
x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest
  • Took 15 min 36 sec . View more details here
  • Description: mage pythonIntegTest
Error signal
  • Took 0 min 0 sec . View more details here
  • Description: Error 'hudson.AbortException: script returned exit code 2'

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Genuine test errors 4

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

  • Name: test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0
  • Name: test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0
  • Name: test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1
  • Name: test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

Merge branch 'mergify/bp/7.17/pr-34315' into mergify/bp/7.17/pr-35780
@cmacknz cmacknz requested a review from a team as a code owner June 20, 2023 23:02
@cmacknz cmacknz requested review from ChrsMark and constanca-m and removed request for a team June 20, 2023 23:02
@cmacknz
Copy link
Member

cmacknz commented Jun 20, 2023

Merged #34355 into this one. We need to upgrade the CI workers to ubuntu:22 so the version of Python matches locally and in the test containers.

This commit improves the compatibility and stability of Python tests
for Filebeat. It addresses two main probmes:
1. Some deprecated TLS functions are now raising deprecation
exceptions (since Python 3.10.0)
2. Some tests, on failure, leave Filebeat running and bound to port
8080, this creates a cascading failure of the subsequent tests that
require the same port to be available.

The main changes are:
 - Ensuring Filebeat process is always terminated, even if the test
 fails
 - Using the replacement methods/objects for the deprecated TLS
 functions.

(cherry picked from commit 682045a)

# Conflicts:
#	libbeat/tests/system/beat/beat.py
@cmacknz
Copy link
Member

cmacknz commented Jun 21, 2023

The filebeat failures need the backport of #35844

@cmacknz
Copy link
Member

cmacknz commented Jun 21, 2023

Merged the changes from backport PR #35844 into this one.

@cmacknz
Copy link
Member

cmacknz commented Jun 21, 2023

The failing v1.18.19 k8s step doesn't seem to actually exist.

@cmacknz
Copy link
Member

cmacknz commented Jun 21, 2023

/test

@cmacknz cmacknz disabled auto-merge June 21, 2023 13:43
@cmacknz
Copy link
Member

cmacknz commented Jun 21, 2023

Merging, only the activemq tests are failing which is happening on other branches now.

@cmacknz cmacknz merged commit 205b27e into 7.17 Jun 21, 2023
90 of 96 checks passed
@cmacknz cmacknz deleted the mergify/bp/7.17/pr-35780 branch June 21, 2023 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport conflicts There is a conflict in the backported pull request needs_team Indicates that the issue/PR needs a Team:* label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants