Batch race condition fix #2630

RafalSkolasinski · 2020-11-12T12:03:16Z

What this PR does / why we need it:

Fixes race condition that sometimes occur in batch processor.
Under certain situation batch processor was exiting before it read input data or before all output data has been written.

This PR fixes it.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

None

review-notebook-app · 2020-11-12T12:03:21Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

seldondev · 2020-11-12T12:03:45Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign rafalskolasinski
You can assign the PR to them by writing /assign @rafalskolasinski in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…utput file

seldondev · 2020-11-12T12:05:05Z

Thu Nov 12 12:05:04 UTC 2020
The logs for [lint] [2] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2630/2.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2630 --build=2

seldondev · 2020-11-12T12:05:05Z

Thu Nov 12 12:05:04 UTC 2020
The logs for [pr-build] [1] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2630/1.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2630 --build=1

axsaucedo · 2020-11-12T12:06:00Z

python/seldon_core/batch_processor.py

+        write_counter += 1
+
+
+def print_counters():


it may be useful part of logging, right now they are very limited so at least having information how many lines were read and written out may be helpful

seldondev · 2020-11-12T12:06:30Z

Thu Nov 12 12:06:25 UTC 2020
The logs for [pr-build] [3] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2630/3.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2630 --build=3

seldondev · 2020-11-12T12:06:37Z

Thu Nov 12 12:06:35 UTC 2020
The logs for [lint] [4] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2630/4.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2630 --build=4

axsaucedo · 2020-11-12T12:06:40Z

python/seldon_core/batch_processor.py

@@ -98,13 +132,14 @@ def _start_input_file_worker(q_in: Queue, input_data_path: str) -> None:
    for line in input_data_file:
        unique_id = str(uuid.uuid1())
        q_in.put((enum_idx, unique_id, line))
+        increment_read()


RafalSkolasinski · 2020-11-12T13:53:29Z

/test integration

seldondev · 2020-11-12T13:54:28Z

Thu Nov 12 13:54:27 UTC 2020
The logs for [pr-build] [5] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2630/5.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2630 --build=5

seldondev · 2020-11-12T13:54:34Z

Thu Nov 12 13:54:33 UTC 2020
The logs for [lint] [6] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2630/6.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2630 --build=6

seldondev · 2020-11-12T13:54:57Z

Thu Nov 12 13:54:56 UTC 2020
The logs for [integration] [7] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2630/7.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2630 --build=7

RafalSkolasinski · 2020-11-12T14:03:27Z

python/seldon_core/batch_processor.py

+
+            counter += 1
+            if counter % 100 == 0:
+                print(f"Processed instances: {counter}")


Not sure if this should be somehow configurable...

RafalSkolasinski · 2020-11-12T14:07:51Z

/test integration

seldondev · 2020-11-12T14:08:44Z

Thu Nov 12 14:08:43 UTC 2020
The logs for [pr-build] [8] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2630/8.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2630 --build=8

seldondev · 2020-11-12T14:08:58Z

Thu Nov 12 14:08:57 UTC 2020
The logs for [lint] [9] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2630/9.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2630 --build=9

seldondev · 2020-11-12T14:09:15Z

Thu Nov 12 14:09:14 UTC 2020
The logs for [integration] [10] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2630/10.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2630 --build=10

RafalSkolasinski · 2020-11-12T14:56:43Z

Integration failure in TestPrepack.test_sklearn_v2

seldondev · 2020-11-12T15:59:13Z

Thu Nov 12 15:59:12 UTC 2020
The logs for [lint] [12] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2630/12.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2630 --build=12

seldondev · 2020-11-12T15:59:14Z

Thu Nov 12 15:59:13 UTC 2020
The logs for [pr-build] [11] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2630/11.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2630 --build=11

seldondev · 2020-11-12T16:34:26Z

Thu Nov 12 16:34:25 UTC 2020
The logs for [pr-build] [13] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2630/13.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2630 --build=13

seldondev · 2020-11-12T16:34:26Z

Thu Nov 12 16:34:25 UTC 2020
The logs for [lint] [14] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2630/14.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2630 --build=14

RafalSkolasinski · 2020-11-13T15:26:59Z

/test integration

seldondev · 2020-11-13T15:28:21Z

Fri Nov 13 15:28:20 UTC 2020
The logs for [integration] [15] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2630/15.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2630 --build=15

seldondev · 2020-11-13T16:02:22Z

@RafalSkolasinski: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
integration	`97d15eb`	link	`/test integration`

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. I understand the commands that are listed here.

RafalSkolasinski · 2020-11-13T16:06:06Z

Only integration failure on TestPrepack.test_sklearn_v2 ,
Unless there is anything to fix, we should merge it.

seldondev added the size/XXL label Nov 12, 2020

fix race conditions that sometimes lead to empty / incomplete batch o…

405aa62

…utput file

seldondev added size/M and removed size/XXL labels Nov 12, 2020

axsaucedo reviewed Nov 12, 2020

View reviewed changes

log every 100th successfully processed data in batch processor

aceb65e

RafalSkolasinski commented Nov 12, 2020

View reviewed changes

avoid using global in batch processor

4d3e11f

use logging instead of print in batch processor

3671b7f

make linter happy

97d15eb

axsaucedo merged commit 0765b3b into SeldonIO:master Nov 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Batch race condition fix #2630

Batch race condition fix #2630

RafalSkolasinski commented Nov 12, 2020

review-notebook-app bot commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

axsaucedo Nov 12, 2020

RafalSkolasinski Nov 12, 2020

RafalSkolasinski Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

axsaucedo Nov 12, 2020

RafalSkolasinski Nov 12, 2020

RafalSkolasinski commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

RafalSkolasinski Nov 12, 2020

RafalSkolasinski commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

RafalSkolasinski commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

RafalSkolasinski commented Nov 13, 2020

seldondev commented Nov 13, 2020

seldondev commented Nov 13, 2020

RafalSkolasinski commented Nov 13, 2020

Batch race condition fix #2630

Batch race condition fix #2630

Conversation

RafalSkolasinski commented Nov 12, 2020

review-notebook-app bot commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

axsaucedo Nov 12, 2020

Choose a reason for hiding this comment

RafalSkolasinski Nov 12, 2020

Choose a reason for hiding this comment

RafalSkolasinski Nov 12, 2020

Choose a reason for hiding this comment

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

axsaucedo Nov 12, 2020

Choose a reason for hiding this comment

RafalSkolasinski Nov 12, 2020

Choose a reason for hiding this comment

RafalSkolasinski commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

RafalSkolasinski Nov 12, 2020

Choose a reason for hiding this comment

RafalSkolasinski commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

RafalSkolasinski commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

seldondev commented Nov 12, 2020

RafalSkolasinski commented Nov 13, 2020

seldondev commented Nov 13, 2020

seldondev commented Nov 13, 2020

RafalSkolasinski commented Nov 13, 2020