Skip to content

Prefix the queue/stage order#421

Merged
mthrok merged 1 commit intomainfrom
tweak-logging
Mar 5, 2025
Merged

Prefix the queue/stage order#421
mthrok merged 1 commit intomainfrom
tweak-logging

Conversation

@mthrok
Copy link
Copy Markdown
Collaborator

@mthrok mthrok commented Mar 5, 2025

So that it's easier to check the stats in the log.

import logging

from spdl.pipeline import PipelineBuilder

logging.basicConfig(level=logging.INFO)

pipeline = (
    PipelineBuilder()
    .add_source(range(10))
    .pipe(lambda x: x+1, name="add1")
    .pipe(lambda x: 2 * x, name="x2")
    .aggregate(3)
    .pipe(sum)
    .aggregate(2)
    .disaggregate()
    .add_sink()
    .build(num_threads=1)
)
with pipeline.auto_stop():
    for _ in pipeline:
        pass
INFO:spdl.pipeline._queue:[0:src_queue]			Processed    10 items in    2.6 [ ms] (QPS: 3809.5) Ave wait time: put:    0.1 [ ms], get (by next stage):    0.0 [ ms].
INFO:spdl.pipeline._queue:[1:add1_queue]		Processed    10 items in    2.9 [ ms] (QPS: 3432.4) Ave wait time: put:    0.0 [ ms], get (by next stage):    0.0 [ ms].
INFO:spdl.pipeline._queue:[2:x2_queue]			Processed    10 items in    3.3 [ ms] (QPS: 3018.1) Ave wait time: put:    0.0 [ ms], get (by next stage):    0.1 [ ms].
INFO:spdl.pipeline._queue:[3:aggregate(3)_queue]	Processed    11 items in    3.8 [ ms] (QPS: 2889.0) Ave wait time: put:    0.0 [ ms], get (by next stage):    0.2 [ ms].
INFO:spdl.pipeline._queue:[4:sum_queue]			Processed     4 items in    4.0 [ ms] (QPS:  998.1) Ave wait time: put:    0.0 [ ms], get (by next stage):    0.7 [ ms].
INFO:spdl.pipeline._queue:[5:aggregate(2)_queue]	Processed     5 items in    4.5 [ ms] (QPS: 1102.1) Ave wait time: put:    0.0 [ ms], get (by next stage):    0.6 [ ms].
INFO:spdl.pipeline._queue:[6:disaggregate_queue]	Processed     4 items in    4.6 [ ms] (QPS:  871.2) Ave wait time: put:    0.0 [ ms], get (by next stage):    1.1 [ ms].
INFO:spdl.pipeline._queue:[7:sink_queue]		Processed     4 items in    4.6 [ ms] (QPS:  863.2) Ave wait time: put:    0.0 [ ms], get (by next stage):    0.9 [ ms].

INFO:spdl.pipeline._hook:[1:add1]		Completed    10 tasks (  0 failed). (Concurrency:   1). Ave task time:    0.1 [ ms].
INFO:spdl.pipeline._hook:[2:x2]			Completed    10 tasks (  0 failed). (Concurrency:   1). Ave task time:    0.1 [ ms].
INFO:spdl.pipeline._hook:[3:aggregate(3)]	Completed    11 tasks (  0 failed). (Concurrency:   1). Ave task time:    0.1 [ ms].
INFO:spdl.pipeline._hook:[4:sum]		Completed     4 tasks (  0 failed). (Concurrency:   1). Ave task time:    0.2 [ ms].
INFO:spdl.pipeline._hook:[5:aggregate(2)]	Completed     5 tasks (  0 failed). (Concurrency:   1). Ave task time:    0.1 [ ms].
INFO:spdl.pipeline._hook:[6:disaggregate]	Completed     4 tasks (  0 failed). (Concurrency:   1). Ave task time:    0.2 [ ms].

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Mar 5, 2025
@facebook-github-bot
Copy link
Copy Markdown
Contributor

@facebook-github-bot has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. (Because this pull request was imported automatically, there will not be any future comments.)

@mthrok mthrok marked this pull request as ready for review March 5, 2025 17:03
@mthrok mthrok changed the title Prefix the queue/stage so that it's easier in logging Prefix the queue/stage order Mar 5, 2025
@mthrok mthrok merged commit f953cad into main Mar 5, 2025
17 of 18 checks passed
@mthrok mthrok deleted the tweak-logging branch March 5, 2025 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants