Upgrade to Go 1.19.10 and handle changes in golang Debian base image #35780

cmacknz · 2023-06-15T13:41:47Z

The plain netcat package appears to have been removed in the latest golang:1.19.10 base image which uses Debian 12 vs Debian 11 which was used previously.

It also appears there have been some changes to Python to work though.

The plain netcat package appears to have been removed in the latest golang:1.19.10 base image.

elasticmachine · 2023-06-15T13:41:50Z

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

mergify · 2023-06-15T13:42:39Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @cmacknz? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2023-06-15T14:56:03Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-06-20T17:51:42.089+0000
Duration: 73 min 31 sec

Test stats 🧪

Test	Results
Failed	4
Passed	27445
Skipped	2013
Total	29462

Test errors

Expand to view the tests failures

`Build&Test / x-pack/metricbeat-pythonIntegTest / test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0`

Expand to view the error details

 Failed: Timeout >90.0s

Expand to view the stacktrace

 self = <test_activemq.ActiveMqTest_0 testMethod=test_queue_metrics_collected>

    @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, 'integration test')
    def test_queue_metrics_collected(self):
>       self.verify_destination_metrics_collection('queue')

module/activemq/test_activemq.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
module/activemq/test_activemq.py:70: in verify_destination_metrics_collection
    conn.disconnect()
/opt/venv/lib/python3.11/site-packages/stomp/connect.py:185: in disconnect
    self.transport.stop()
/opt/venv/lib/python3.11/site-packages/stomp/transport.py:122: in stop
    self.__receiver_thread_exit_condition.wait()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Condition(<unlocked _thread.RLock object owner=0 count=0 at 0x7f31871dfec0>, 0)>
timeout = None

    def wait(self, timeout=None):
        """Wait until notified or until a timeout occurs.
    
        If the calling thread has not acquired the lock when this method is
        called, a RuntimeError is raised.
    
        This method releases the underlying lock, and then blocks until it is
        awakened by a notify() or notify_all() call for the same condition
        variable in another thread, or until the optional timeout occurs. Once
        awakened or timed out, it re-acquires the lock and returns.
    
        When the timeout argument is present and not None, it should be a
        floating point number specifying a timeout for the operation in seconds
        (or fractions thereof).
    
        When the underlying lock is an RLock, it is not released using its
        release() method, since this may not actually unlock the lock when it
        was acquired multiple times recursively. Instead, an internal interface
        of the RLock class is used, which really unlocks it even when it has
        been recursively acquired several times. Another internal interface is
        then used to restore the recursion level when the lock is reacquired.
    
        """
        if not self._is_owned():
            raise RuntimeError("cannot wait on un-acquired lock")
        waiter = _allocate_lock()
        waiter.acquire()
        self._waiters.append(waiter)
        saved_state = self._release_save()
        gotit = False
        try:    # restore state no matter what (e.g., KeyboardInterrupt)
            if timeout is None:
>               waiter.acquire()
E               Failed: Timeout >90.0s

/usr/lib/python3.11/threading.py:320: Failed

`Build&Test / x-pack/metricbeat-pythonIntegTest / test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0`

Expand to view the error details

 Failed: Timeout >90.0s

Expand to view the stacktrace

 self = <test_activemq.ActiveMqTest_0 testMethod=test_topic_metrics_collected>

    @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, 'integration test')
    def test_topic_metrics_collected(self):
>       self.verify_destination_metrics_collection('topic')

module/activemq/test_activemq.py:100: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
module/activemq/test_activemq.py:70: in verify_destination_metrics_collection
    conn.disconnect()
/opt/venv/lib/python3.11/site-packages/stomp/connect.py:185: in disconnect
    self.transport.stop()
/opt/venv/lib/python3.11/site-packages/stomp/transport.py:122: in stop
    self.__receiver_thread_exit_condition.wait()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Condition(<unlocked _thread.RLock object owner=0 count=0 at 0x7f3186de3dc0>, 0)>
timeout = None

    def wait(self, timeout=None):
        """Wait until notified or until a timeout occurs.
    
        If the calling thread has not acquired the lock when this method is
        called, a RuntimeError is raised.
    
        This method releases the underlying lock, and then blocks until it is
        awakened by a notify() or notify_all() call for the same condition
        variable in another thread, or until the optional timeout occurs. Once
        awakened or timed out, it re-acquires the lock and returns.
    
        When the timeout argument is present and not None, it should be a
        floating point number specifying a timeout for the operation in seconds
        (or fractions thereof).
    
        When the underlying lock is an RLock, it is not released using its
        release() method, since this may not actually unlock the lock when it
        was acquired multiple times recursively. Instead, an internal interface
        of the RLock class is used, which really unlocks it even when it has
        been recursively acquired several times. Another internal interface is
        then used to restore the recursion level when the lock is reacquired.
    
        """
        if not self._is_owned():
            raise RuntimeError("cannot wait on un-acquired lock")
        waiter = _allocate_lock()
        waiter.acquire()
        self._waiters.append(waiter)
        saved_state = self._release_save()
        gotit = False
        try:    # restore state no matter what (e.g., KeyboardInterrupt)
            if timeout is None:
>               waiter.acquire()
E               Failed: Timeout >90.0s

/usr/lib/python3.11/threading.py:320: Failed

`Build&Test / x-pack/metricbeat-pythonIntegTest / test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1`

Expand to view the error details

 Failed: Timeout >90.0s

Expand to view the stacktrace

 self = <test_activemq.ActiveMqTest_1 testMethod=test_queue_metrics_collected>

    @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, 'integration test')
    def test_queue_metrics_collected(self):
>       self.verify_destination_metrics_collection('queue')

module/activemq/test_activemq.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
module/activemq/test_activemq.py:70: in verify_destination_metrics_collection
    conn.disconnect()
/opt/venv/lib/python3.11/site-packages/stomp/connect.py:185: in disconnect
    self.transport.stop()
/opt/venv/lib/python3.11/site-packages/stomp/transport.py:122: in stop
    self.__receiver_thread_exit_condition.wait()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Condition(<unlocked _thread.RLock object owner=0 count=0 at 0x7f3187131440>, 0)>
timeout = None

    def wait(self, timeout=None):
        """Wait until notified or until a timeout occurs.
    
        If the calling thread has not acquired the lock when this method is
        called, a RuntimeError is raised.
    
        This method releases the underlying lock, and then blocks until it is
        awakened by a notify() or notify_all() call for the same condition
        variable in another thread, or until the optional timeout occurs. Once
        awakened or timed out, it re-acquires the lock and returns.
    
        When the timeout argument is present and not None, it should be a
        floating point number specifying a timeout for the operation in seconds
        (or fractions thereof).
    
        When the underlying lock is an RLock, it is not released using its
        release() method, since this may not actually unlock the lock when it
        was acquired multiple times recursively. Instead, an internal interface
        of the RLock class is used, which really unlocks it even when it has
        been recursively acquired several times. Another internal interface is
        then used to restore the recursion level when the lock is reacquired.
    
        """
        if not self._is_owned():
            raise RuntimeError("cannot wait on un-acquired lock")
        waiter = _allocate_lock()
        waiter.acquire()
        self._waiters.append(waiter)
        saved_state = self._release_save()
        gotit = False
        try:    # restore state no matter what (e.g., KeyboardInterrupt)
            if timeout is None:
>               waiter.acquire()
E               Failed: Timeout >90.0s

/usr/lib/python3.11/threading.py:320: Failed

`Build&Test / x-pack/metricbeat-pythonIntegTest / test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1`

Expand to view the error details

 Failed: Timeout >90.0s

Expand to view the stacktrace

 self = <test_activemq.ActiveMqTest_1 testMethod=test_topic_metrics_collected>

    @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, 'integration test')
    def test_topic_metrics_collected(self):
>       self.verify_destination_metrics_collection('topic')

module/activemq/test_activemq.py:100: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
module/activemq/test_activemq.py:70: in verify_destination_metrics_collection
    conn.disconnect()
/opt/venv/lib/python3.11/site-packages/stomp/connect.py:185: in disconnect
    self.transport.stop()
/opt/venv/lib/python3.11/site-packages/stomp/transport.py:122: in stop
    self.__receiver_thread_exit_condition.wait()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Condition(<unlocked _thread.RLock object owner=0 count=0 at 0x7f318878a840>, 0)>
timeout = None

    def wait(self, timeout=None):
        """Wait until notified or until a timeout occurs.
    
        If the calling thread has not acquired the lock when this method is
        called, a RuntimeError is raised.
    
        This method releases the underlying lock, and then blocks until it is
        awakened by a notify() or notify_all() call for the same condition
        variable in another thread, or until the optional timeout occurs. Once
        awakened or timed out, it re-acquires the lock and returns.
    
        When the timeout argument is present and not None, it should be a
        floating point number specifying a timeout for the operation in seconds
        (or fractions thereof).
    
        When the underlying lock is an RLock, it is not released using its
        release() method, since this may not actually unlock the lock when it
        was acquired multiple times recursively. Instead, an internal interface
        of the RLock class is used, which really unlocks it even when it has
        been recursively acquired several times. Another internal interface is
        then used to restore the recursion level when the lock is reacquired.
    
        """
        if not self._is_owned():
            raise RuntimeError("cannot wait on un-acquired lock")
        waiter = _allocate_lock()
        waiter.acquire()
        self._waiters.append(waiter)
        saved_state = self._release_save()
        gotit = False
        try:    # restore state no matter what (e.g., KeyboardInterrupt)
            if timeout is None:
>               waiter.acquire()
E               Failed: Timeout >90.0s

/usr/lib/python3.11/threading.py:320: Failed

Steps errors

Expand to view the steps failures

`x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest`

Took 18 min 21 sec . View more details here
Description: mage pythonIntegTest

`x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest`

Took 12 min 10 sec . View more details here
Description: mage pythonIntegTest

`x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest`

Took 12 min 25 sec . View more details here
Description: mage pythonIntegTest

`Error signal`

Took 0 min 0 sec . View more details here
Description: Error 'hudson.AbortException: script returned exit code 1'

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Build&Test / x-pack/metricbeat-pythonIntegTest / test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0
Name: Build&Test / x-pack/metricbeat-pythonIntegTest / test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0
Name: Build&Test / x-pack/metricbeat-pythonIntegTest / test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1
Name: Build&Test / x-pack/metricbeat-pythonIntegTest / test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

cmacknz · 2023-06-15T14:59:31Z

Looks like there is more than one problem, going to revert to Go 1.19.10 version bump to fix this faster and then iterate on this PR separately.

This fixes the PEP668 error about "externally managed environments" which prevents modifying the system python installation.

…35751)" (elastic#35784)" This reverts commit 27b1799.

shmsr

Left minor comments. LGTM otherwise!

auditbeat/Dockerfile

CHANGELOG.next.asciidoc

Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com>

Remove dependency on pytest-otel as it prevented upgrading pytest.

cmacknz · 2023-06-19T19:26:20Z

libbeat/tests/system/requirements.txt

 pycodestyle==2.6.0
 pyparsing==2.4.7
 pyrsistent==0.16.0
-pytest==7.1.3


Upgraded along with py to fix pytest-dev/apipkg#30

I removed the pytest-otel dependency since it was pinning us to pytest 7.1.3 and wouldn't let me upgrade. I'm not sure it is strictly necessary anyway.

Apparently we own the pytest-otel package, I think we need to loosen the dependency to require pytest >= 7.1.3 instead of pytest == 7.1.3 if we can. FYI @kuisathaverat

https://github.com/elastic/apm-pipeline-library/blob/cd076539fb75e75ca1a7fe740e5fac0abbf569f8/resources/scripts/pytest_otel/pyproject.toml#L5

dependencies = [ "opentelemetry-api==1.15.0", "opentelemetry-exporter-otlp==1.15.0", "opentelemetry-sdk==1.15.0", "pytest==7.1.3", ]

You can remove the pytest-otel, your team is not using the OpenTelemetry data the plugin reports.

Remove duplicate dependencies from the docker files.

cmacknz · 2023-06-20T19:06:01Z

/test x-pack/metricbeat-pythonIntegTest

…35780) * Explicitly install netcat-openbsd everywhere. The plain netcat package appears to have been removed in the latest golang:1.19.10 base image. * Setup a virtualenv in Docker containers. This fixes the PEP668 error about "externally managed environments" which prevents modifying the system python installation. * Revert "Revert "[Automation] Bump Golang version to 1.19.10 (#35751)" (#35784)" This reverts commit 27b1799. * Reword CHANGELOG.next.asciidoc Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com> * Improve PEP668 comments. * Install latest PyYAML in test containers. * Add check to see if already in a venv. * Upgrade pytest and py. Remove dependency on pytest-otel as it prevented upgrading pytest. * Check for pre-set virtualenv earlier. * Always create a fresh virtualenv in Docker. Remove duplicate dependencies from the docker files. * Fix metricbeat integration tests. * Force install Python dependencies for Metricbeat. --------- Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com> (cherry picked from commit 62374dd) # Conflicts: # dev-tools/mage/pytest.go # libbeat/docs/version.asciidoc # libbeat/tests/system/requirements.txt # libbeat/tests/system/requirements_aix.txt # metricbeat/Dockerfile

…35780) * Explicitly install netcat-openbsd everywhere. The plain netcat package appears to have been removed in the latest golang:1.19.10 base image. * Setup a virtualenv in Docker containers. This fixes the PEP668 error about "externally managed environments" which prevents modifying the system python installation. * Revert "Revert "[Automation] Bump Golang version to 1.19.10 (#35751)" (#35784)" This reverts commit 27b1799. * Reword CHANGELOG.next.asciidoc Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com> * Improve PEP668 comments. * Install latest PyYAML in test containers. * Add check to see if already in a venv. * Upgrade pytest and py. Remove dependency on pytest-otel as it prevented upgrading pytest. * Check for pre-set virtualenv earlier. * Always create a fresh virtualenv in Docker. Remove duplicate dependencies from the docker files. * Fix metricbeat integration tests. * Force install Python dependencies for Metricbeat. --------- Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com> (cherry picked from commit 62374dd) # Conflicts: # libbeat/docs/version.asciidoc

elasticmachine · 2023-06-20T20:01:56Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-06-20T19:06:12.246+0000
Duration: 56 min 32 sec

Test stats 🧪

Test	Results
Failed	4
Passed	21
Skipped	5
Total	30

Test errors

Expand to view the tests failures

`test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0`

Expand to view the error details

 Failed: Timeout >90.0s

Expand to view the stacktrace

 self = <test_activemq.ActiveMqTest_0 testMethod=test_queue_metrics_collected>

    @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, 'integration test')
    def test_queue_metrics_collected(self):
>       self.verify_destination_metrics_collection('queue')

module/activemq/test_activemq.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
module/activemq/test_activemq.py:70: in verify_destination_metrics_collection
    conn.disconnect()
/opt/venv/lib/python3.11/site-packages/stomp/connect.py:185: in disconnect
    self.transport.stop()
/opt/venv/lib/python3.11/site-packages/stomp/transport.py:122: in stop
    self.__receiver_thread_exit_condition.wait()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Condition(<unlocked _thread.RLock object owner=0 count=0 at 0x7f1addaa77c0>, 0)>
timeout = None

    def wait(self, timeout=None):
        """Wait until notified or until a timeout occurs.
    
        If the calling thread has not acquired the lock when this method is
        called, a RuntimeError is raised.
    
        This method releases the underlying lock, and then blocks until it is
        awakened by a notify() or notify_all() call for the same condition
        variable in another thread, or until the optional timeout occurs. Once
        awakened or timed out, it re-acquires the lock and returns.
    
        When the timeout argument is present and not None, it should be a
        floating point number specifying a timeout for the operation in seconds
        (or fractions thereof).
    
        When the underlying lock is an RLock, it is not released using its
        release() method, since this may not actually unlock the lock when it
        was acquired multiple times recursively. Instead, an internal interface
        of the RLock class is used, which really unlocks it even when it has
        been recursively acquired several times. Another internal interface is
        then used to restore the recursion level when the lock is reacquired.
    
        """
        if not self._is_owned():
            raise RuntimeError("cannot wait on un-acquired lock")
        waiter = _allocate_lock()
        waiter.acquire()
        self._waiters.append(waiter)
        saved_state = self._release_save()
        gotit = False
        try:    # restore state no matter what (e.g., KeyboardInterrupt)
            if timeout is None:
>               waiter.acquire()
E               Failed: Timeout >90.0s

/usr/lib/python3.11/threading.py:320: Failed

`test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0`

Expand to view the error details

 Failed: Timeout >90.0s

Expand to view the stacktrace

 self = <test_activemq.ActiveMqTest_0 testMethod=test_topic_metrics_collected>

    @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, 'integration test')
    def test_topic_metrics_collected(self):
>       self.verify_destination_metrics_collection('topic')

module/activemq/test_activemq.py:100: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
module/activemq/test_activemq.py:70: in verify_destination_metrics_collection
    conn.disconnect()
/opt/venv/lib/python3.11/site-packages/stomp/connect.py:185: in disconnect
    self.transport.stop()
/opt/venv/lib/python3.11/site-packages/stomp/transport.py:122: in stop
    self.__receiver_thread_exit_condition.wait()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Condition(<unlocked _thread.RLock object owner=0 count=0 at 0x7f1adc3a94c0>, 0)>
timeout = None

    def wait(self, timeout=None):
        """Wait until notified or until a timeout occurs.
    
        If the calling thread has not acquired the lock when this method is
        called, a RuntimeError is raised.
    
        This method releases the underlying lock, and then blocks until it is
        awakened by a notify() or notify_all() call for the same condition
        variable in another thread, or until the optional timeout occurs. Once
        awakened or timed out, it re-acquires the lock and returns.
    
        When the timeout argument is present and not None, it should be a
        floating point number specifying a timeout for the operation in seconds
        (or fractions thereof).
    
        When the underlying lock is an RLock, it is not released using its
        release() method, since this may not actually unlock the lock when it
        was acquired multiple times recursively. Instead, an internal interface
        of the RLock class is used, which really unlocks it even when it has
        been recursively acquired several times. Another internal interface is
        then used to restore the recursion level when the lock is reacquired.
    
        """
        if not self._is_owned():
            raise RuntimeError("cannot wait on un-acquired lock")
        waiter = _allocate_lock()
        waiter.acquire()
        self._waiters.append(waiter)
        saved_state = self._release_save()
        gotit = False
        try:    # restore state no matter what (e.g., KeyboardInterrupt)
            if timeout is None:
>               waiter.acquire()
E               Failed: Timeout >90.0s

/usr/lib/python3.11/threading.py:320: Failed

`test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1`

Expand to view the error details

 Failed: Timeout >90.0s

Expand to view the stacktrace

 self = <test_activemq.ActiveMqTest_1 testMethod=test_queue_metrics_collected>

    @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, 'integration test')
    def test_queue_metrics_collected(self):
>       self.verify_destination_metrics_collection('queue')

module/activemq/test_activemq.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
module/activemq/test_activemq.py:70: in verify_destination_metrics_collection
    conn.disconnect()
/opt/venv/lib/python3.11/site-packages/stomp/connect.py:185: in disconnect
    self.transport.stop()
/opt/venv/lib/python3.11/site-packages/stomp/transport.py:122: in stop
    self.__receiver_thread_exit_condition.wait()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Condition(<unlocked _thread.RLock object owner=0 count=0 at 0x7f1adc8e2d40>, 0)>
timeout = None

    def wait(self, timeout=None):
        """Wait until notified or until a timeout occurs.
    
        If the calling thread has not acquired the lock when this method is
        called, a RuntimeError is raised.
    
        This method releases the underlying lock, and then blocks until it is
        awakened by a notify() or notify_all() call for the same condition
        variable in another thread, or until the optional timeout occurs. Once
        awakened or timed out, it re-acquires the lock and returns.
    
        When the timeout argument is present and not None, it should be a
        floating point number specifying a timeout for the operation in seconds
        (or fractions thereof).
    
        When the underlying lock is an RLock, it is not released using its
        release() method, since this may not actually unlock the lock when it
        was acquired multiple times recursively. Instead, an internal interface
        of the RLock class is used, which really unlocks it even when it has
        been recursively acquired several times. Another internal interface is
        then used to restore the recursion level when the lock is reacquired.
    
        """
        if not self._is_owned():
            raise RuntimeError("cannot wait on un-acquired lock")
        waiter = _allocate_lock()
        waiter.acquire()
        self._waiters.append(waiter)
        saved_state = self._release_save()
        gotit = False
        try:    # restore state no matter what (e.g., KeyboardInterrupt)
            if timeout is None:
>               waiter.acquire()
E               Failed: Timeout >90.0s

/usr/lib/python3.11/threading.py:320: Failed

`test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1`

Expand to view the error details

 Failed: Timeout >90.0s

Expand to view the stacktrace

 self = <test_activemq.ActiveMqTest_1 testMethod=test_topic_metrics_collected>

    @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, 'integration test')
    def test_topic_metrics_collected(self):
>       self.verify_destination_metrics_collection('topic')

module/activemq/test_activemq.py:100: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
module/activemq/test_activemq.py:70: in verify_destination_metrics_collection
    conn.disconnect()
/opt/venv/lib/python3.11/site-packages/stomp/connect.py:185: in disconnect
    self.transport.stop()
/opt/venv/lib/python3.11/site-packages/stomp/transport.py:122: in stop
    self.__receiver_thread_exit_condition.wait()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Condition(<unlocked _thread.RLock object owner=0 count=0 at 0x7f1adc4318c0>, 0)>
timeout = None

    def wait(self, timeout=None):
        """Wait until notified or until a timeout occurs.
    
        If the calling thread has not acquired the lock when this method is
        called, a RuntimeError is raised.
    
        This method releases the underlying lock, and then blocks until it is
        awakened by a notify() or notify_all() call for the same condition
        variable in another thread, or until the optional timeout occurs. Once
        awakened or timed out, it re-acquires the lock and returns.
    
        When the timeout argument is present and not None, it should be a
        floating point number specifying a timeout for the operation in seconds
        (or fractions thereof).
    
        When the underlying lock is an RLock, it is not released using its
        release() method, since this may not actually unlock the lock when it
        was acquired multiple times recursively. Instead, an internal interface
        of the RLock class is used, which really unlocks it even when it has
        been recursively acquired several times. Another internal interface is
        then used to restore the recursion level when the lock is reacquired.
    
        """
        if not self._is_owned():
            raise RuntimeError("cannot wait on un-acquired lock")
        waiter = _allocate_lock()
        waiter.acquire()
        self._waiters.append(waiter)
        saved_state = self._release_save()
        gotit = False
        try:    # restore state no matter what (e.g., KeyboardInterrupt)
            if timeout is None:
>               waiter.acquire()
E               Failed: Timeout >90.0s

/usr/lib/python3.11/threading.py:320: Failed

Steps errors

Expand to view the steps failures

`x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest`

Took 19 min 0 sec . View more details here
Description: mage pythonIntegTest

`x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest`

Took 12 min 26 sec . View more details here
Description: mage pythonIntegTest

`x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest`

Took 12 min 26 sec . View more details here
Description: mage pythonIntegTest

`Error signal`

Took 0 min 0 sec . View more details here
Description: Error 'hudson.AbortException: script returned exit code 1'

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0
Name: test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0
Name: test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1
Name: test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

…lang Debian base image (#35841) * Upgrade to Go 1.19.10 and handle changes in golang Debian base image (#35780) * Explicitly install netcat-openbsd everywhere. The plain netcat package appears to have been removed in the latest golang:1.19.10 base image. * Setup a virtualenv in Docker containers. This fixes the PEP668 error about "externally managed environments" which prevents modifying the system python installation. * Revert "Revert "[Automation] Bump Golang version to 1.19.10 (#35751)" (#35784)" This reverts commit 27b1799. * Reword CHANGELOG.next.asciidoc Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com> * Improve PEP668 comments. * Install latest PyYAML in test containers. * Add check to see if already in a venv. * Upgrade pytest and py. Remove dependency on pytest-otel as it prevented upgrading pytest. * Check for pre-set virtualenv earlier. * Always create a fresh virtualenv in Docker. Remove duplicate dependencies from the docker files. * Fix metricbeat integration tests. * Force install Python dependencies for Metricbeat. --------- Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com> (cherry picked from commit 62374dd) # Conflicts: # libbeat/docs/version.asciidoc * Fix conflict in version.asciidoc * Remove extra entry in CHANGELOG.next.asciidoc --------- Co-authored-by: Craig MacKenzie <craig.mackenzie@elastic.co>

…olang Debian base image (#35840) * ci: use ubuntu:22 workers (#34315) (cherry picked from commit a5f15b6) # Conflicts: # .ci/beats-tester.groovy # .ci/build-docker-images.groovy # .ci/heartbeat-synthetics.groovy # x-pack/filebeat/Jenkinsfile.yml # x-pack/metricbeat/Jenkinsfile.yml * remove unexisting files in 7.17 when backporting another PR * fix merge conflicts * Upgrade to Go 1.19.10 and handle changes in golang Debian base image (#35780) * Explicitly install netcat-openbsd everywhere. The plain netcat package appears to have been removed in the latest golang:1.19.10 base image. * Setup a virtualenv in Docker containers. This fixes the PEP668 error about "externally managed environments" which prevents modifying the system python installation. * Revert "Revert "[Automation] Bump Golang version to 1.19.10 (#35751)" (#35784)" This reverts commit 27b1799. * Reword CHANGELOG.next.asciidoc Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com> * Improve PEP668 comments. * Install latest PyYAML in test containers. * Add check to see if already in a venv. * Upgrade pytest and py. Remove dependency on pytest-otel as it prevented upgrading pytest. * Check for pre-set virtualenv earlier. * Always create a fresh virtualenv in Docker. Remove duplicate dependencies from the docker files. * Fix metricbeat integration tests. * Force install Python dependencies for Metricbeat. --------- Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com> (cherry picked from commit 62374dd) # Conflicts: # dev-tools/mage/pytest.go # libbeat/docs/version.asciidoc # libbeat/tests/system/requirements.txt # libbeat/tests/system/requirements_aix.txt # metricbeat/Dockerfile * Remove extra entry in CHANGELOG.next.asciidoc * Fix conflict in version.asciidoc * Fix conflict in requirements.txt * Fix conflict in requirements_aix.txt * Fix conflict in metricbeat Dockerfile * Fix conflict in pytest.go * Add missing force create virtual env option. * python tests: compatibility and stability improvements (#31588) This commit improves the compatibility and stability of Python tests for Filebeat. It addresses two main probmes: 1. Some deprecated TLS functions are now raising deprecation exceptions (since Python 3.10.0) 2. Some tests, on failure, leave Filebeat running and bound to port 8080, this creates a cascading failure of the subsequent tests that require the same port to be available. The main changes are: - Ensuring Filebeat process is always terminated, even if the test fails - Using the replacement methods/objects for the deprecated TLS functions. (cherry picked from commit 682045a) # Conflicts: # libbeat/tests/system/beat/beat.py * Remove duplicate entries in CHANGELOG-developer.next.asciidoc * Resolve conflicts in beat.py --------- Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> Co-authored-by: Craig MacKenzie <craig.mackenzie@elastic.co> Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

Explicitly install netcat-openbsd everywhere.

9833ba4

The plain netcat package appears to have been removed in the latest golang:1.19.10 base image.

cmacknz added the Team:Elastic-Agent Label for the Agent team label Jun 15, 2023

cmacknz self-assigned this Jun 15, 2023

cmacknz requested review from a team as code owners June 15, 2023 13:41

cmacknz requested review from belimawr and faec June 15, 2023 13:41

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 15, 2023

rdner approved these changes Jun 15, 2023

View reviewed changes

This was referenced Jun 15, 2023

Revert "[Automation] Bump Golang version to 1.19.10 (#35751)" #35784

Merged

Revert "[8.8](backport #35751) [Automation] Bump Golang version to 1.… #35785

Merged

Setup a virtualenv in Docker containers.

b821dff

This fixes the PEP668 error about "externally managed environments" which prevents modifying the system python installation.

cmacknz added the backport-v8.8.0 Automated backport with mergify label Jun 15, 2023

cmacknz added 2 commits June 15, 2023 14:25

Merge branch 'main' into fix-netcat-apt-install

3fd06e6

Revert "Revert "[Automation] Bump Golang version to 1.19.10 (elastic#…

2f04444

…35751)" (elastic#35784)" This reverts commit 27b1799.

cmacknz changed the title ~~Explicitly install netcat-openbsd everywhere.~~ Upgrade to Go 1.19.10 and handle changes in golang Debian base image Jun 15, 2023

shmsr reviewed Jun 15, 2023

View reviewed changes

auditbeat/Dockerfile Outdated Show resolved Hide resolved

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved

cmacknz and others added 2 commits June 15, 2023 15:36

Reword CHANGELOG.next.asciidoc

7b50519

Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com>

Improve PEP668 comments.

49c526e

shmsr approved these changes Jun 15, 2023

View reviewed changes

Install latest PyYAML in test containers.

df12af0

cmacknz force-pushed the fix-netcat-apt-install branch from 595d74e to df12af0 Compare June 16, 2023 23:50

pierrehilbert mentioned this pull request Jun 19, 2023

Build 106 for 8.8 with status FAILURE #35776

Closed

cmacknz added 2 commits June 19, 2023 14:45

Add check to see if already in a venv.

c0f9088

Upgrade pytest and py.

3c1c398

Remove dependency on pytest-otel as it prevented upgrading pytest.

cmacknz commented Jun 19, 2023

View reviewed changes

Check for pre-set virtualenv earlier.

461865a

belimawr approved these changes Jun 20, 2023

View reviewed changes

Always create a fresh virtualenv in Docker.

8fff6ff

Remove duplicate dependencies from the docker files.

cmacknz added the backport-v7.17.0 Automated backport with mergify label Jun 20, 2023

cmacknz added 2 commits June 20, 2023 12:48

Fix metricbeat integration tests.

a2348eb

Force install Python dependencies for Metricbeat.

169fe58

cmacknz merged commit 62374dd into elastic:main Jun 20, 2023
99 of 101 checks passed

cmacknz deleted the fix-netcat-apt-install branch June 20, 2023 19:29

This was referenced Jun 20, 2023

[7.17](backport #35780) Upgrade to Go 1.19.10 and handle changes in golang Debian base image #35840

Merged

[8.8](backport #35780) Upgrade to Go 1.19.10 and handle changes in golang Debian base image #35841

Merged

reakaleek mentioned this pull request Jul 19, 2023

Fix ironbank validation in 8.8 #36115

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to Go 1.19.10 and handle changes in golang Debian base image #35780

Upgrade to Go 1.19.10 and handle changes in golang Debian base image #35780

cmacknz commented Jun 15, 2023 •

edited

Loading

elasticmachine commented Jun 15, 2023

mergify bot commented Jun 15, 2023

elasticmachine commented Jun 15, 2023 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

`Build&Test / x-pack/metricbeat-pythonIntegTest / test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0`

`Build&Test / x-pack/metricbeat-pythonIntegTest / test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0`

`Build&Test / x-pack/metricbeat-pythonIntegTest / test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1`

`Build&Test / x-pack/metricbeat-pythonIntegTest / test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1`

`x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest`

`x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest`

`x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest`

`Error signal`

Genuine test errors

cmacknz commented Jun 15, 2023

shmsr left a comment

cmacknz Jun 19, 2023 •

edited

Loading

kuisathaverat Jun 20, 2023

cmacknz commented Jun 20, 2023

elasticmachine commented Jun 20, 2023

Build stats

Test stats 🧪

`test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0`

`test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0`

`test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1`

`test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1`

`x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest`

`x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest`

`x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest`

`Error signal`

Genuine test errors

Upgrade to Go 1.19.10 and handle changes in golang Debian base image #35780

Upgrade to Go 1.19.10 and handle changes in golang Debian base image #35780

Conversation

cmacknz commented Jun 15, 2023 • edited Loading

elasticmachine commented Jun 15, 2023

mergify bot commented Jun 15, 2023

elasticmachine commented Jun 15, 2023 • edited by jenkins-beats-ci bot Loading

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

Build&Test / x-pack/metricbeat-pythonIntegTest / test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0

Build&Test / x-pack/metricbeat-pythonIntegTest / test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0

Build&Test / x-pack/metricbeat-pythonIntegTest / test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1

Build&Test / x-pack/metricbeat-pythonIntegTest / test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1

Steps errors

x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest

x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest

x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest

Error signal

🐛 Flaky test report

Genuine test errors

🤖 GitHub comments

cmacknz commented Jun 15, 2023

shmsr left a comment

Choose a reason for hiding this comment

cmacknz Jun 19, 2023 • edited Loading

Choose a reason for hiding this comment

kuisathaverat Jun 20, 2023

Choose a reason for hiding this comment

cmacknz commented Jun 20, 2023

elasticmachine commented Jun 20, 2023

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0

test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0

test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1

test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1

Steps errors

x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest

x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest

x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest

Error signal

🐛 Flaky test report

Genuine test errors

🤖 GitHub comments

cmacknz commented Jun 15, 2023 •

edited

Loading

elasticmachine commented Jun 15, 2023 •

edited by jenkins-beats-ci bot

Loading

`Build&Test / x-pack/metricbeat-pythonIntegTest / test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0`

`Build&Test / x-pack/metricbeat-pythonIntegTest / test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0`

`Build&Test / x-pack/metricbeat-pythonIntegTest / test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1`

`Build&Test / x-pack/metricbeat-pythonIntegTest / test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1`

`x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest`

`x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest`

`x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest`

`Error signal`

cmacknz Jun 19, 2023 •

edited

Loading

`test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0`

`test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0`

`test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1`

`test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1`

`x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest`

`x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest`

`x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest`

`Error signal`