Cgi/verifier/scripts for addional tests #4584

nieznanysprawiciel · 2019-08-05T16:13:48Z

This PR introduces following changes:

Scripts for extended tests of blender verifier
Semi integrations tests utilities
Tests reproducing some known bugs in blender verifier

Detailed description:

Scripts are checking different subtasks splits on predefined checker scene
- Tests check if verifier computes crops coordinates correctly and if crops are correctly matched with verified image region.
- Tests don't use verifier verdict, but instead compare produced crops pixel by pixel with reference crops. This bypasses problem with not ideal verification quality.
- Scripts execution lasts very long, so they are not included in standard test set. They will be ran once and all discovered bugs will be reproduced using semi-integrations tests. This means that scripts don't affect Golem in any way.
Semi integrations tests utilities:
- Test utilities help write tests running full application flow (I mean golem integrations from app directory).
- Utils use TaskManager class to emulate golem behavior as close as possible.
- Tests don't need network and external nodes. Providers are mocked on local computer.
- keep_testdir_on_fail decorator was added to TempDirFixture, to easier check content of test temporary directories, when debugging.
- These utils are used by transcoding branches as well, so merging them will help us to easier merge to develop in the future.
Tests reproducing some known bugs in blender verifier
- Test reproducing bugs found so far by verifier scripts.
- Reproduced bugs are already fixed on develop, this PR adds only reproduction that uses semi-integration tests utils.

…_subtask

apps/blender/resources/images/entrypoints/scripts/verifier_tools/verifier.py

golem/testutils_app_integration.py

golem/verifier/blender_verifier.py

tests/apps/blender/task/test_blenderintegration.py

tests/golem/verifier/blenderverifier_ext_test_script.py

Co-Authored-By: Adam Mizerski <amizerski@golem.network>

shadeofblue · 2019-10-29T08:41:28Z

golem/testutils.py

+# Use this decorator if you don't want to remove test directory
+# after test execution.
+# This is debug decorator - don't remove even if it isn't used.
+def keep_testdir_on_fail(fun):


wouldn't it be better to have an optional command line argument instead of a decorator for this purpose instead?

I prefere this solution, because cmdline switch would enable it for all test cases. They produce a lot of garbage and is is a lot harder to find results of only one specific test

golem/testutils_app_integration.py

shadeofblue · 2019-10-29T08:48:45Z

golem/testutils_app_integration.py

+        self.assertFalse(timeouted)
+        return self.task_manager.verify_subtask(subtask_id)
+
+    def _execute_subtask(self, task: Task, ctd: dict):


is there a reason some of those methods here are public and some are private? are any of them called from outside the test class?

Some of them are designed to be called outside of TestTaskIntegration and some of them are not. But not all of them are used yet.

etam

shadeofblue

nieznanysprawiciel added 30 commits July 24, 2019 15:43

Pick TestTaskIntegration from CGI/transocoding/master

75da6bc

Migration fix

a147c39

Preserve changes to testutils from develop

a7a6b98

Separate blender_verifier parameters generation

eacb503

Implemented local integration test for blender (test dont pass)

202396c

Blender integration test passes

504a07c

Ignore .vscode

3964d96

Integration tests don't place output in golem directory

a9741ef

Hack reactor restart

9d13b5a

test_blenderintegration works with reactor

09aa7be

Split execute method into start_task, compute_next_subtask and verify…

2e17764

…_subtask

Blender integration test uses chessboard

7539459

blender integration test computes multiple frames

9a5be38

Verifier testing scripts - skeleton

72bf291

Move scripts to separate file

a8af2e7

Move scripts to separate file

4fb98c7

Fix blender integration test for multiple frames

ab8e1b1

Script - reporting classes

0bc59e0

Catching and reporting exceptions

8250532

Replication of blender integration bug

85d43d1

Generate different provider id for each subtask to avoid ban

b55bd0b

Better stack trace in report

a3fdd80

Write tests results to files

664ef88

Tests can be run using parameters from outside

6dfdb78

Verifier script uses code for comparing crops

9929d4c

Functions organized better to provide more useful functionalities

953bda0

Separate TestTaskIntegration

2db7950

Read crop parameters from log

fe7fbff

Add crops data to reports

66e4db3

Report fails for subtasks and merge reports

7ba73d4

nieznanysprawiciel and others added 3 commits October 24, 2019 15:45

Fix some directory bugs in integration tests

5d8b76b

Fix linter part 1

dc03a84

Fix linter part 2 - unused import

34a6d08

nieznanysprawiciel requested review from shadeofblue and etam October 24, 2019 15:28

etam suggested changes Oct 25, 2019

View reviewed changes

nieznanysprawiciel and others added 8 commits October 25, 2019 12:36

Apply suggestions from code review

c85d41a

Co-Authored-By: Adam Mizerski <amizerski@golem.network>

Apply suggestions from code review

bfdb70e

Co-Authored-By: Adam Mizerski <amizerski@golem.network>

Remove check_file_existence

ae6b373

Remove _mock_remove_files

7e2c47c

use dataclass

4dd7f9e

json.load(file) review change

7a2c560

blender_verifier docker_task initialization style fix

c442502

Remove get_report function from blender verifier extended scripts

d140b95

shadeofblue reviewed Oct 29, 2019

View reviewed changes

golem/testutils_app_integration.py Outdated Show resolved Hide resolved

shadeofblue reviewed Oct 29, 2019

View reviewed changes

Daniel Bigos added 4 commits October 29, 2019 16:01

fix: proper result wrapping

e3f811e

refactor: use json library instead of pformat in verifier scripts

eb215ce

Merge branch 'develop' into CGI/verifier/scripts-for-addional-tests

622c877

chore: linter

d950a7f

bidzyyys force-pushed the CGI/verifier/scripts-for-addional-tests branch from cadbd19 to d950a7f Compare October 31, 2019 16:28

bidzyyys requested a review from etam November 2, 2019 10:52

etam approved these changes Nov 4, 2019

View reviewed changes

Update blender_verifier image

d1f7fab

shadeofblue approved these changes Nov 5, 2019

View reviewed changes

nieznanysprawiciel added 2 commits November 5, 2019 13:51

Fix dumping float32 error

ce01d3a

Fix linter

2a4dc11

magdasta merged commit 55ba1ef into develop Nov 5, 2019

bidzyyys deleted the CGI/verifier/scripts-for-addional-tests branch November 5, 2019 14:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cgi/verifier/scripts for addional tests #4584

Cgi/verifier/scripts for addional tests #4584

nieznanysprawiciel commented Aug 5, 2019 •

edited

Loading

shadeofblue Oct 29, 2019

nieznanysprawiciel Oct 29, 2019

shadeofblue Oct 29, 2019

nieznanysprawiciel Oct 29, 2019

etam left a comment

shadeofblue left a comment

Cgi/verifier/scripts for addional tests #4584

Cgi/verifier/scripts for addional tests #4584

Conversation

nieznanysprawiciel commented Aug 5, 2019 • edited Loading

This PR introduces following changes:

Detailed description:

shadeofblue Oct 29, 2019

Choose a reason for hiding this comment

nieznanysprawiciel Oct 29, 2019

Choose a reason for hiding this comment

shadeofblue Oct 29, 2019

Choose a reason for hiding this comment

nieznanysprawiciel Oct 29, 2019

Choose a reason for hiding this comment

etam left a comment

Choose a reason for hiding this comment

shadeofblue left a comment

Choose a reason for hiding this comment

nieznanysprawiciel commented Aug 5, 2019 •

edited

Loading