Integration test & unit tests jobs separation #6649

auvipy · 2021-02-28T05:15:01Z

No description provided.

auvipy · 2021-02-28T05:25:07Z

@xirdneh you can push improvement on this branch as use this as a base. I am checking how service containers going to work here for integration tests

auvipy · 2021-02-28T05:52:06Z

pypy tests were failing before this PR.

codecov · 2021-02-28T05:56:26Z

Codecov Report

Merging #6649 (a1ae742) into master (3cf5072) will decrease coverage by 0.15%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #6649      +/-   ##
==========================================
- Coverage   89.23%   89.08%   -0.16%     
==========================================
  Files         138      138              
  Lines       16623    16623              
  Branches     2099     2099              
==========================================
- Hits        14834    14808      -26     
- Misses       1570     1597      +27     
+ Partials      219      218       -1

Flag	Coverage Δ
unittests	`89.07% <ø> (-0.16%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
celery/backends/couchbase.py	`40.62% <0.00%> (-42.19%)`	⬇️
celery/backends/asynchronous.py	`60.64% <0.00%> (+0.46%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d3e5df3...a1ae742. Read the comment docs.

auvipy · 2021-02-28T05:59:00Z

redis connection is throwing errors. the service container might need some adjsutment

thedrow · 2021-02-28T10:58:04Z

.github/workflows/python-package.yml

@@ -63,3 +64,71 @@ jobs:
            flags: unittests # optional
            fail_ci_if_error: true # optional (default = false)
            verbose: true # optional (default = false)
+
+
+  Integration:


We need to set the integration tests to only run after the unit tests to save resources.
See syntax: https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idneeds

.github/workflows/python-package.yml

auvipy · 2021-03-05T04:30:59Z

@matusvalo please suggest modification too.

matusvalo · 2021-03-22T20:49:20Z

Let me check CI in celery. It should be pretty straightforward to port kombu/py-amqp CI to celery also...

maybe-sybr · 2021-05-23T23:34:37Z

Just freshened this on master. I think I'll spend a little time on it today to see if I can get the redis suite quite working as a first step and then the rabbitmq/rpc prior to merge

maybe-sybr · 2021-05-24T00:20:07Z

The suite starts running now, but we have a busted test which fails the run after a few retries.

lgtm-com · 2021-05-24T01:05:54Z

This pull request introduces 1 alert when merging 1dac185 into 025bad6 - view on LGTM.com

new alerts:

1 for Unused import

maybe-sybr · 2021-05-24T01:47:01Z

t/integration/test_inspect.py

@@ -117,7 +117,7 @@ def test_active(self, inspect):
                    'exchange': '',
                    'routing_key': 'celery',
                    'priority': 0,
-                    'redelivered': False
+                    'redelivered': ANY,


I'm not sure why these tests stopped passing. AFAICT the redelivered seems to be set to None implying that we "don't know" (or don't bother to default to False) if a task has been redelivered or not. These tests /probably/ (??) don't care about that so changing this to an ANY seems like it is okay.

I'd like for someone a bit more familiar with redelivery logic to think about this if possible. I just don't know what we should be expecting here and why it's not False like it must have been in the past.

redelivered is set when the broker redelivers the message (at least with AMQP brokers).
I honestly don't have an idea what has changed. Do you mind debugging this?

I'll see if I can find some time later this week

lgtm-com · 2021-05-24T02:04:22Z

This pull request introduces 1 alert and fixes 2 when merging bfab125 into 025bad6 - view on LGTM.com

new alerts:

1 for Unused import

fixed alerts:

1 for Non-exception in 'except' clause
1 for Module is imported with 'import' and 'import from'

maybe-sybr · 2021-05-24T03:46:47Z

Not sure why that inspect test is failing. It passes on my box reliably. Leaving this diff for the moment, someone else can feel free to take a look at it in the meantime.

lgtm-com · 2021-05-24T03:49:36Z

This pull request introduces 1 alert when merging 8f8d402 into 025bad6 - view on LGTM.com

new alerts:

1 for Unused import

maybe-sybr · 2021-05-24T03:50:19Z

.github/workflows/python-package.yml

+        python-version: ['3.7', ] #'3.8', '3.9', 'pypy3']
+        # XXX: Does each matrix variation get their own copy of the service
+        # containers? Test cross-talk may be an issue...
+        toxenv: ['redis', ] #'rabbitmq']


This should probably actually be called backend since it's only the tail of the toxenv name.

thedrow · 2021-08-08T14:12:01Z

I rebased this PR.
We have to merge this soon so we can work on 5.2 safely.

lgtm-com · 2021-08-08T14:50:15Z

This pull request fixes 2 alerts when merging 762668e into d3e5df3 - view on LGTM.com

fixed alerts:

2 for Wrong name for an argument in a call

lgtm-com · 2021-08-09T10:49:54Z

This pull request fixes 2 alerts when merging a1ae742 into d3e5df3 - view on LGTM.com

fixed alerts:

2 for Wrong name for an argument in a call

thedrow · 2021-08-09T14:04:59Z

What is wrong with the test suite?
Is it running locally just fine? @celery/core-developers

maybe-sybr · 2021-08-11T00:30:50Z

     File "/home/runner/work/celery/celery/celery/app/trace.py", line 358, in build_tracer
      push_request = request_stack.push
  AttributeError: 'NoneType' object has no attribute 'push'

This seems pretty bad 😬

auvipy · 2021-11-11T11:38:03Z

     File "/home/runner/work/celery/celery/celery/app/trace.py", line 358, in build_tracer
      push_request = request_stack.push
  AttributeError: 'NoneType' object has no attribute 'push'

This seems pretty bad grimacing

was the rabbitmq tests passing? I cant remember, I can only remember redis was failing. I will be doing some experiment in another PR to see if rabbit services & tests are running. I goose step at a time, lets see

auvipy added this to the 5.1.0 milestone Feb 28, 2021

auvipy requested a review from xirdneh February 28, 2021 05:23

thedrow requested changes Feb 28, 2021

View reviewed changes

thedrow added this to In progress in Celery 5.1.0 via automation Mar 7, 2021

thedrow added Category: Project Governance PR Type: Enhancement labels Mar 7, 2021

maybe-sybr mentioned this pull request May 18, 2021

test: Improve use of test fixtures in integration tests #6773

Closed

thedrow moved this from In progress to Postponed in Celery 5.1.0 May 23, 2021

maybe-sybr force-pushed the ci branch from 6f99337 to 2f576ce Compare May 23, 2021 23:15

maybe-sybr force-pushed the ci branch 4 times, most recently from 33d5350 to a080602 Compare May 24, 2021 00:06

maybe-sybr force-pushed the ci branch from a080602 to 1dac185 Compare May 24, 2021 00:35

maybe-sybr force-pushed the ci branch from 1dac185 to bfab125 Compare May 24, 2021 01:44

maybe-sybr reviewed May 24, 2021

View reviewed changes

thedrow removed this from the 5.1.0 milestone May 24, 2021

auvipy and others added 9 commits August 8, 2021 17:08

Integration test & unit tests jobs separation

df5d3f5

rabbitmq service container

84f5531

disbale couchbase temporarily

9ed6e05

redis service container

3fdac3c

redis port

8210dff

redis host to localhost

ed47bfd

DO NOT MERGE: hack: Use tox for redis integration suite

3320eba

ci: Add -vv to pytest integration test cmd

93659df

squash! DO NOT MERGE: hack: Use tox for redis integration suite

762668e

thedrow force-pushed the ci branch from ada7d5a to 762668e Compare August 8, 2021 14:11

thedrow added this to the 5.2 milestone Aug 8, 2021

thedrow added 2 commits August 9, 2021 12:31

Fix rebase blunder.

12a67c2

Increase integration jobs timeout.

a1ae742

thedrow mentioned this pull request Sep 23, 2021

replaced_task_nesting property should have an integration test #6968

Open

auvipy modified the milestones: 5.2, 5.2.x Oct 31, 2021

auvipy self-assigned this Oct 31, 2021

auvipy mentioned this pull request Jan 11, 2022

[WIP] - intial tox-docker config for dockerised integration tests #7216

Draft

dobosevych mentioned this pull request Apr 11, 2022

Binary can't be passed as argument #7448

Open

17 tasks

auvipy mentioned this pull request Apr 14, 2022

Integration test fix #7460

Merged

auvipy modified the milestones: 5.2.x, 5.3 Apr 23, 2022

auvipy closed this Oct 12, 2022

auvipy deleted the ci branch June 2, 2023 00:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integration test & unit tests jobs separation #6649

Integration test & unit tests jobs separation #6649

auvipy commented Feb 28, 2021

auvipy commented Feb 28, 2021

auvipy commented Feb 28, 2021

codecov bot commented Feb 28, 2021 •

edited

auvipy commented Feb 28, 2021

thedrow Feb 28, 2021

auvipy commented Mar 5, 2021

matusvalo commented Mar 22, 2021

maybe-sybr commented May 23, 2021

maybe-sybr commented May 24, 2021

lgtm-com bot commented May 24, 2021

maybe-sybr May 24, 2021

thedrow May 24, 2021

maybe-sybr May 24, 2021

lgtm-com bot commented May 24, 2021

maybe-sybr commented May 24, 2021 •

edited

lgtm-com bot commented May 24, 2021

maybe-sybr May 24, 2021

thedrow commented Aug 8, 2021

lgtm-com bot commented Aug 8, 2021

lgtm-com bot commented Aug 9, 2021

thedrow commented Aug 9, 2021

maybe-sybr commented Aug 11, 2021

auvipy commented Nov 11, 2021

Integration test & unit tests jobs separation #6649

Integration test & unit tests jobs separation #6649

Conversation

auvipy commented Feb 28, 2021

auvipy commented Feb 28, 2021

auvipy commented Feb 28, 2021

codecov bot commented Feb 28, 2021 • edited

Codecov Report

auvipy commented Feb 28, 2021

thedrow Feb 28, 2021

Choose a reason for hiding this comment

auvipy commented Mar 5, 2021

matusvalo commented Mar 22, 2021

maybe-sybr commented May 23, 2021

maybe-sybr commented May 24, 2021

lgtm-com bot commented May 24, 2021

maybe-sybr May 24, 2021

Choose a reason for hiding this comment

thedrow May 24, 2021

Choose a reason for hiding this comment

maybe-sybr May 24, 2021

Choose a reason for hiding this comment

lgtm-com bot commented May 24, 2021

maybe-sybr commented May 24, 2021 • edited

lgtm-com bot commented May 24, 2021

maybe-sybr May 24, 2021

Choose a reason for hiding this comment

thedrow commented Aug 8, 2021

lgtm-com bot commented Aug 8, 2021

lgtm-com bot commented Aug 9, 2021

thedrow commented Aug 9, 2021

maybe-sybr commented Aug 11, 2021

auvipy commented Nov 11, 2021

codecov bot commented Feb 28, 2021 •

edited

maybe-sybr commented May 24, 2021 •

edited