Remove @pytest.mark.quarantined from stable quarantine tests

[Taragolis]

related: #28658

Currently all tests pass either local run or quarantined pipeline in CI. I've just remove flag from all test for check in all possible backends.

Most of the issues in the past related to MySQL 5.7 which almost reach EOL, however quarantined pipeline run only with SQLite backend.

Update
After couple runs we found that only tests/executors/test_celery_executor.py::TestLocalTaskJob::test_process_sigterm_works_with_retries is still flaky

[potiuk]

Cool. I think we shoudl do it a "bit" gradually (and let them run for a few days after merge). This will be helpful to narrow down the problem in case some of the tests introduce side-effect, to be able to determine which ones it is more easily.

[potiuk]

Rebased them to re-run and re-check if they are repeatably stable as well.

[potiuk]

And rebased them with public runners too.

[potiuk]

@Taragolis -> https://github.com/apache/airflow/actions/runs/3864325055/jobs/6587088003 - seems that at least test_process_sigterm_works_with_retries is flaky on public runners ...

[Taragolis]

@potiuk I hope it just flaky test and not something wrong with callback mechanism.
Let keep it for a while and have a look how often it failed - just for some statistic, I will exclude this test anyway and try to figure out what the actual nature of this failure.

[Taragolis]

I can't reproduce this error locally but I have an idea what could possible wrong

Potentially callback might acquire lock

airflow/tests/jobs/test_local_task_job.py

Lines 779 to 784 in ce67786

	retry_callback_called = Value("i", 0)
	def retry_callback(context):
	with retry_callback_called.get_lock():
	retry_callback_called.value += 1
	assert context["dag_run"].dag_id == "test_mark_failure_2"

After it checked in test case

airflow/tests/jobs/test_local_task_job.py

Lines 803 to 805 in ce67786

	with timeout(10):
	job1.run()
	assert retry_callback_called.value == 1

However locally I do not have this behaviour. I only could emulate it by adding time.sleep(5) before acquire lock in callback.

[Taragolis]

And ... this test failed again 😞

[potiuk]

Yeah. I think we could merge the others and leave that one as the last - I might also be able to take a look at it shortly (this is kinda what I was expecting and that's why I wanted to do it gradually) :)

[Taragolis]

Yeah, let me revert quarantined mark for this test.

I just think about what we tried to test here:

1. Is on_retry_callback actually called
   or
2. Is task retried, e.g. change state to up_for_retry

[potiuk]

One more rebase just in case...

[Taragolis]

@potiuk looks like this tests stable. We would know more about stability after we merge it 🤣

[potiuk]

Let's