ZENKO-1016 Improvement: Add tagging to flaky node e2e tests

[tmacro]

To help improve the CI, add tags to flaky test. These test will be skipped during normal test runs, to help with the endless rebuilding, and will be run during the nightly build to prevent loss of test coverage.

[bert-e]

Hello tmacro,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get
information on this process.

Status report is not available.

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• 2 peers

Peer approvals must include at least 1 approval from the following list:

• @ssalaues

• @vrancurel

• @giacomoguiulfo

• @NicolasT

[bert-e]

Build failed

The build for commit did not succeed in branch improvement/ZENKO-1016_Add_tagging_to_node_e2e_tests.

[bennettbuchanan]

I agree that we should not be slowed by test flakiness, so it makes sense that we take measures to address it. For example, previously we had been adding retries as a strategy to address this problem. However, my concern would be that running only once may conceal a larger problem with the functionality. That is, if something is merged during the day that actually breaks this test entirely (or increases it's flakiness), then when this test fails in the evening it would be easy to dismiss it as test flakiness. Partly the question is: do we know why tagged test are flaky, and if not, how will we follow up to make the test suite reliable?

[bennettbuchanan]

Seeing only an added line in this file, so wondering if we meant to add a tag to these tests as well.

[ssalaues]

@bennettbuchanan you're right we should be able to track these flaky tests instead of losing them to the ether. Also the idea is that we're going to run the nightly tests several times in attempt to measure the flakiness.

@tmacro can you create a ticket for the flaky tests and add a comment to the tagged test pointing to the ticket?

[tmacro]

@bennettbuchanan the envisioned workflow is, after a test is tagged flaky, a matching ticket is created to either fix the flakiness, or discover the bug causing it. This way the tests won't stay "flaky" for very long, nor be swept under the rug as someone is actively investigating it.

[bert-e]

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

• ✔️ development/0.9

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of Release Engineering will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

• Any commit you add on the source branch will trigger a new cycle after the
  current queue is merged.
• Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of Release Engineering now.

[bert-e]

I have successfully merged the changeset of this pull request
into targetted development branches:

• ✔️ development/0.9

Please check the status of the associated issue ZENKO-1016.

Goodbye tmacro.