Automatic demands #313

mat-hek · 2021-06-22T12:35:53Z

closes #285

The idea

This PR adds support for the automatic handling of demands in filters. The idea is to specify relationships between input and output pads, and instead of calling handle_demand callback and handling demand and redemand actions, automatically calculate and forward the demands between the related pads. The profits of that approach are the following:

user won't have to be aware of demands if they don't need to affect the flow control
by not calling the callbacks nor handling actions we can achieve better performance

The implementation

The implementation does not introduce any breaking changes to the API

Relaying demands

Let's assume we have the following pipeline:

A -> B -> C

working in pull mode, and B has auto demands turned on. The flow from B's perspective goes as follows:

a) initially, B sends demand of arbitrary size (hardcoded, at least for now) to A

b) when B receives a demand from C, it stores it in state and checks whether there's some demand on A remaining

if not, then it sends an identical demand as in (a)

c) when B receives buffers from A, it subtracts their size from the demand on A and checks if it's still positive

if not, then it checks if there's some demand from C remaining
- if so, then it sends an identical demand as in (a)
no matter what, it immediately passes new buffers to handle_process, which usually sends buffers to C (but may queue or ignore them as well)
if any buffers are sent, then B subtracts their size from C's demand

Because of checking the output demand before buffers are processed, we can avoid making the processing synchronous. On the other hand, by sending buffers immediately, we avoid using input buffer, which brings some overhead. That may theoretically lead to generating many buffers before slowing down the stream in some specific cases. If that becomes a problem, we may want to try falling back to the input buffer in such cases.

Atomic toilet

Since the new demanding mode doesn't use the input buffer, we had to solve the toilet separately. On the other hand, the toilet in the input buffer had a serious problem. Let's consider a pipeline:

A -> [toilet] B -> C

If the C slows down, and B doesn't receive enough demands from it, while A still sends data, the toilet will overflow. However, if B slows down a lot and hangs on processing a few buffers for a long time, the new buffers can stuck in the mailbox and the toilet won't be aware of them.

The only way of solving that I have figured out was to move the responsibility to the sender - A in this case. Each time the sender sends a buffer, it increments the toilet size and checks if it doesn't exceed the limit. If it does, A kills the receiver - B. On the receiver side, the toilet size is decremented upon each received buffer. That's possible without any message exchange, thanks to erlang's atomics.

Toilet overflow detection algorithm

That brings us to the toilet overflow detection algorithm. Currently, the toilet has a hard size limit, above which it crashes. It's difficult to provide a good default for it though. It's also not easy to configure it properly. We should think over how to solve it better - possibly we should see how the size behaves in some small period of time and decide whether to fail based on that. We could also make it possible to configure whether to kill the sender or the receiver in case of overflow.

Bin linking strategy

Sharing the toilet atomics and configuration between elements within different bins occurred to be difficult and error-prone in the current state of things. It was a clear sign that it's about the time to refactor that, possibly avoiding bins proxying media across their boundaries. Unfortunately, it turned out to be not an easy task. Here's how it's solved now:

The user code returns children & links spec to the parent. The parent validates and resolves it.
The parent sends link requests to the affected children.
- When a bin receives a link request, it executes handle_pad_added (dynamic pads) and binds the external pad to the internal one. This usually entails some internal linking, so link requests are sent too. When link responses for all link requests are received, the bin responds to all link requests it received.
- When an element receives a link request, it responds immediately.
When children respond to all link requests triggered by the spec, the actual linking takes place. For each link, the parent GenServer.calls owner of the output pad, which in turn calls the owner of the input pad. Bins proxy that calls according to the bindings from the link request phase.
When linking is finished, the parent initiates playback change of the newly created children.
If linking is not finished in 5 seconds, an error is raised.

This solution allows bins to spawn children or do any asynchronous work when a pad is added and makes the links established directly between elements. Thanks to that, we could get rid of the bin's linking buffer and media proxying in bins.

To do

Relaying demands
Integration with manual demands
Handling corner cases, like adding and removing pads
Media streams are sent directly between elements even if they're in different bins
Atomics-based toilet
Tests
Docs & type specs

Further work (issues to be created):

Toilet overflow detection & toilet configuration
Performance optimization of integration with manual demands
Dynamically switching between manual and automatic demands
API for defining relationships between pads

sax

Here is a first pass through this. I still don't have a full understanding of the link lifecycle and consequences of internal vs external links, so will try to get through that later today.

sax · 2021-10-27T18:15:49Z

lib/membrane/core/element/demand_controller.ex

+  defp auto_demands_positive?(pad_ref, state) do
+    PadModel.get_data!(state, pad_ref, :demand_pads)
+    |> Enum.all?(&(PadModel.get_data!(state, &1, :demand) > 0))
+  end
+


As a defp only used in this file, where the entire pad data is retrieved unchanged in the caller, it would be safe and slightly more efficient to just pass in the pad data directly.

sax · 2021-10-27T18:16:31Z

lib/membrane/core/element/demand_controller.ex

+    %{demand: demand, toilet: toilet} = data = PadModel.get_data!(state, pad_ref)
+    demand_size = state.demand_size
+
+    if demand <= div(demand_size, 2) and auto_demands_positive?(pad_ref, state) do


Suggested change

if demand <= div(demand_size, 2) and auto_demands_positive?(pad_ref, state) do

if demand <= div(demand_size, 2) and auto_demands_positive?(data, state) do

sax · 2021-10-27T18:17:05Z

lib/membrane/core/element/demand_controller.ex

+  defp auto_demands_positive?(pad_ref, state) do
+    PadModel.get_data!(state, pad_ref, :demand_pads)


Suggested change

defp auto_demands_positive?(pad_ref, state) do

PadModel.get_data!(state, pad_ref, :demand_pads)

defp auto_demands_positive?(pad_data, state) do

Map.get(pad_data, :demand_pads)

sax · 2021-10-27T18:56:16Z

lib/membrane/core/parent/lifecycle_controller.ex

-      Core.Child.LifecycleController.unlink(state)
+    if state.__struct__ == Membrane.Core.Bin.State do
+      case {old, new} do
+        {:stopped, :perpared} -> Core.Child.PadController.check_for_unlinked_static_pads(state)


Suggested change

{:stopped, :perpared} -> Core.Child.PadController.check_for_unlinked_static_pads(state)

{:stopped, :prepared} -> Core.Child.PadController.check_for_unlinked_static_pads(state)

lib/membrane/core/bin/pad_controller.ex

sax · 2021-10-27T21:34:45Z

lib/membrane/core/element/pad_controller.ex

+  defp check_for_auto_demands(%{mode: :pull, demand_mode: :auto} = pad_data, state) do
+    state =
+      Enum.reduce(pad_data.demand_pads, state, fn pad, state ->
+        PadModel.update_data!(state, pad, :demand_pads, &List.delete(&1, pad_data.ref))
+      end)
+
+    if pad_data.direction == :output do
+      Enum.reduce(pad_data.demand_pads, state, &DemandController.check_auto_demand/2)
+    else
+      state
+    end
+  end


Is there any way this could be collapsed into one reduce? I'm also curious if PadModel.set_data!(state, pad_ref, :demand, demand - buf_size) can be moved into DemandController... it seems like every caller of DemandController.check_auto_demand/2 needs to update demand, then call check_auto_demand, which also updates demand. Could the demand logic account for this so the demand size is only updated once?

I suppose it could, but not sure how to do it to make it look better...

it seems like every caller of DemandController.check_auto_demand/2 needs to update demand

That's actually not true :P

then call check_auto_demand, which also updates demand

It doesn't always do so

Could the demand logic account for this so the demand size is only updated once?

Given the above facts, it's hard to achieve

I'm also curious if PadModel.set_data!(state, pad_ref, :demand, demand - buf_size) can be moved into DemandController

That's a good idea, I moved it

bblaszkow06 · 2021-10-28T15:18:03Z

lib/membrane/child_entry.ex

@@ -33,7 +33,8 @@ defmodule Membrane.ChildEntry do
    :pid,
    :clock,
    :sync,
-    playback_synced?: false,
+    :spec,


There's no such key in the type spec

This is almost always used as spec_ref. Shouldn't that be the name? Plus I'm missing the docs about what it does

lib/membrane/core/bin/pad_controller.ex

bblaszkow06 · 2021-10-28T16:40:04Z

lib/membrane/core/bin/state.ex

                children_log_metadata: [],
-                links: []
+                links: [],
+                pending_specs: %{}


Not reflected in typespec

lib/membrane/core/child/pad_controller.ex

bblaszkow06 · 2021-10-28T16:58:06Z

lib/membrane/core/element.ex

        _from,
        state
      ) do
-    PadController.handle_link(direction, this, other, other_info, state) |> reply(state)
+    Membrane.Logger.debug("handle link")


If this is going to stay (and is not a leftover from testing) I think you should add more context to it i.e. what is connecting to what

bblaszkow06 · 2021-10-28T16:59:54Z

lib/membrane/core/element.ex

+         Message.new(:link_request, [_pad_ref, _direction, link_id, _pad_props]),
+         state
+       ) do
+    Membrane.Logger.debug("link request")


Same as above

andpodob · 2021-11-05T16:05:33Z

lib/membrane/pad_data.ex

@@ -62,5 +62,9 @@ defmodule Membrane.Pad.Data do
            sticky_messages: nil,
            input_buf: nil,
            demand: nil,
-            options: %{}
+            options: %{},
+            demand_pads: [],


WDYT about :demanding_pads, or simply :associated_output_pads? I had a hard time connecting the name of this field with its purpose.

andpodob · 2021-11-08T10:44:59Z

lib/membrane/core/element/demand_controller.ex

    end
  end

+  @spec check_auto_demand(Pad.ref_t(), State.t()) :: State.t()
+  def check_auto_demand(pad_ref, demand_decrease \\ 0, state) do
+    %{demand: demand, toilet: toilet, demand_pads: demand_pads} =


We need to handle EOS by probably removing the pad from demand_pads to make auto_demands_positive? check work properly when one of the pads receives EOS.

bblaszkow06 · 2021-11-08T10:48:59Z

lib/membrane/core/element/demand_controller.ex

+  @spec check_auto_demand(Pad.ref_t(), State.t()) :: State.t()
+  def check_auto_demand(pad_ref, demand_decrease \\ 0, state) do


I think the function allowing to decrease the demand and having side effects shouldn't be called check_X

lib/membrane/core/element/demand_handler.ex

bblaszkow06 · 2021-11-08T12:22:05Z

lib/membrane/core/element/demand_handler.ex

+      when is_reference(toilet) do
+    %{other_demand_unit: other_demand_unit, pid: pid} = data
+    buf_size = Buffer.Metric.from_unit(other_demand_unit).buffers_size(buffers)
+    toilet_size = :atomics.add_get(toilet, 1, buf_size)


I had to check the docs to find out what 1 means here. I'd add a module attribute naming it
EDIT: or even wrap the atomics with a very simple module hiding the implementation details, although that would be an additional call

Erlang API great as always ;) Not sure what is simpler though - this or having a wrapper (which could be a macro)

Anything is better than this. Either do

Suggested change

toilet_size = :atomics.add_get(toilet, 1, buf_size)

toilet_size = :atomics.add_get(toilet, @toilet_size_idx, buf_size)

or create a wrapper

Suggested change

toilet_size = :atomics.add_get(toilet, 1, buf_size)

toilet_size = AtomicCounter.add_get(toilet, buf_size)

EDIT: Another idea - create a wrapper over Toilet and hide the ascii art logging there

bblaszkow06 · 2021-11-08T12:26:40Z

lib/membrane/core/element/demand_handler.ex

+    buf_size = Buffer.Metric.from_unit(other_demand_unit).buffers_size(buffers)
+    toilet_size = :atomics.add_get(toilet, 1, buf_size)
+
+    if toilet_size > 200 do


Module attribute? BTW, isn't too low for demands in bytes?

We're going to replace that with some smart algorithm

I don't see any smart algorithm right now. This value will break the toilet for byte-pushing elements and I don't think it can be merged like this

bblaszkow06 · 2021-11-08T14:57:28Z

lib/membrane/core/element/demand_handler.ex

+        state
+      ) do
+    buf_size = Buffer.Metric.from_unit(other_demand_unit).buffers_size(buffers)
+    PadModel.update_data!(state, pad_ref, :demand, &(&1 - buf_size))


The pair of get_in & put_in seems to be more than 2 times faster than update:

map = %{ list: [], counter: 0, pad: %{ :input => %{cnt: 1}, {:input, 0} => %{cnt: 1} }, and_some: :atom } range = 1..10_000 cases = %{ put: fn -> range |> Enum.reduce(map, fn i, acc -> cnt = acc |> Bunch.Access.get_in([:pad, :input, :cnt]) acc |> Bunch.Access.put_in([:pad, :input, :cnt], cnt + i) end) end, update: fn -> range |> Enum.reduce(map, fn i, acc -> acc |> Bunch.Access.update_in([:pad, :input, :cnt], &(&1 + i)) end) end } Benchee.run(cases)

Operating System: Linux CPU Information: AMD Ryzen 7 PRO 4750U with Radeon Graphics Number of Available Cores: 16 Available memory: 29.11 GB Elixir 1.12.3 Erlang 24.1 Benchmark suite executing with the following configuration: warmup: 2 s time: 5 s memory time: 0 ns parallel: 1 inputs: none specified Estimated total run time: 14 s Benchmarking put... Benchmarking update... | Name | ips | average | deviation | median | 99th % | | put | 27.62 | 36.21 ms | ±2.88% | 35.68 ms | 40.21 ms | | update | 11.25 | 88.85 ms | ±2.56% | 87.69 ms | 95.41 ms | Comparison: put 27.62 update 11.25 - 2.45x slower +52.65 ms

I'd optimize the PadModel itself, but in a separate PR

lib/membrane/core/element/pad_controller.ex

bblaszkow06 · 2021-11-08T18:04:20Z

lib/membrane/core/element/state.ex

@@ -81,7 +82,8 @@ defmodule Membrane.Core.Element.State do
        clock: nil,
        stream_sync: options.sync,
        latency: 0
-      }
+      },
+      demand_size: 4000


Have you considered making it configurable?

I believe it ultimately should somehow automatically adapt.

Right now it doesn't so it should be configurable to have a workaround if the value isn't appropriate

bblaszkow06 · 2021-11-08T18:05:27Z

test/membrane/core/element_test.exs

             accepted_caps: :any,
             availability: :always,
             direction: :output,
             mode: :pull,
             name: :output,
             options: nil
-           }
+           } = reply


Maybe you should rather change the asserted result

bblaszkow06 · 2021-11-08T18:06:05Z

test/membrane/integration/bin_test.exs

            },
            sink: Testing.Sink
          ]
        })

+      Process.sleep(2000)


Don't forget to remove

test/membrane/integration/bin_test.exs

bblaszkow06 · 2022-01-21T16:57:29Z

lib/membrane/parent_spec.ex

-    to `#{Membrane.Core.Element.InputQueue.default_demand_excess_factor()}` (the default may change in the future).
-  - `min_demand_factor` - A factor by which the minimal demand is multiplied by. Used only for pads working in pull
+    in a queue called toilet. If the toilet size grows above its capacity, it overflows by raising an error.
+  - `demand_excess` - Used only for pads working in pull mode with manual demands. See `t:Membrane.Pad.mode_t/0`


The mode_t has invalid link in its description (missing /0 for demand_mode_t)

bblaszkow06 · 2022-01-21T17:00:48Z

lib/membrane/parent_spec.ex

+  - `demand_excess` - Used only for pads working in pull mode with manual demands. See `t:Membrane.Pad.mode_t/0`
+    and `t:Membrane.Pad.demand_mode_t/0` for more info.
+  - `min_demand_factor` - A factor used to calculate minimal demand (`minimal_demand = demand_excess * min_demand_factor`)
+    by which `demand_excess` is multiplied by to calculate the minimal demand. Used only for pads working in pull
    mode with manual demands. See `t:Membrane.Pad.mode_t/0` and `t:Membrane.Pad.demand_mode_t/0` for more info. Defaults
    to `#{Membrane.Core.Element.InputQueue.default_min_demand_factor()}` (the default may change in the future).
-  - `auto_demand_size_factor` - A factor by which the auto demand size multiplied by. Used only for pads working in pull
-    mode with automatic demands. See `t:Membrane.Pad.mode_t/0` and `t:Membrane.Pad.demand_mode_t/0` for more info. Defaults
-    to `#{Membrane.Core.Element.DemandHandler.default_auto_demand_size_factor()}` (the default may change in the future).
+  - `auto_demand_size` - Used only for pads working in pull mode with automatic demands. See `t:Membrane.Pad.mode_t/0`
+    and `t:Membrane.Pad.demand_mode_t/0` for more info.


The docs miss the explanation of what those values do

bblaszkow06 · 2022-01-24T12:25:26Z

lib/membrane/parent_spec.ex

+  - `demand_excess` - Demand that will be generated automatically by Membrane to allow smooth, concurrent processing.
+    All buffers received in response to that demand will be queued internally until they are actually demanded by user.
+    Used only for pads working in pull mode with manual demands. See `t:Membrane.Pad.mode_t/0`


This still doesn't explain everything and only remembering the old preffered_size I could get it. I'd modify the docs and change the name of the option:

Suggested change

- `demand_excess` - Demand that will be generated automatically by Membrane to allow smooth, concurrent processing.

All buffers received in response to that demand will be queued internally until they are actually demanded by user.

Used only for pads working in pull mode with manual demands. See `t:Membrane.Pad.mode_t/0`

- `initial_demand` - The initial demand sent to the linked output pad. It is used to fill the queue of the input pad.

That allows for fulfilling the demands of the element by taking data from the queue while the actual sending of demands is done asynchronously, smoothing the processing.

Used only for pads working in pull mode with manual demands. See `t:Membrane.Pad.mode_t/0`

Ideas for alternative names:

warm_up_demand

starting_demand

This may however affect the way min_demand_factor is defined, I'm not sure if if makes sense to define the min_demand as function of this demand and calculated using a factor

initial demand seems like it was only sent initially and didn't matter afterwards. I refactored the description but left demand_excess

andpodob

LGTM! 🎉

mat-hek self-assigned this Jun 22, 2021

mat-hek added this to In progress in What's happening in Membrane? via automation Jun 22, 2021

mat-hek changed the title ~~auto demand~~ Automatic demands Jun 22, 2021

mat-hek force-pushed the auto-demand branch from 1e69ec6 to 1361a32 Compare June 29, 2021 14:54

mat-hek added the webrtc label Jul 28, 2021

mat-hek moved this from In progress to To do in What's happening in Membrane? Jul 28, 2021

mat-hek force-pushed the auto-demand branch 2 times, most recently from e583440 to 975e084 Compare August 12, 2021 14:47

mat-hek moved this from To do to In progress in What's happening in Membrane? Aug 24, 2021

mat-hek force-pushed the auto-demand branch 2 times, most recently from d446e06 to 97ef055 Compare September 3, 2021 16:54

mat-hek force-pushed the auto-demand branch from 97ef055 to b2feb42 Compare September 7, 2021 14:25

mat-hek force-pushed the auto-demand branch from b2feb42 to e4cff9d Compare September 16, 2021 08:00

mat-hek force-pushed the auto-demand branch from 6239ca2 to c57f3f7 Compare October 14, 2021 14:45

mickel8 removed this from In progress in What's happening in Membrane? Oct 19, 2021

mickel8 modified the milestone: core 0.8.0 Oct 19, 2021

mat-hek force-pushed the auto-demand branch from 1521964 to 8d9a563 Compare October 22, 2021 14:56

mat-hek requested review from bblaszkow06 and andpodob October 27, 2021 12:55

mat-hek marked this pull request as ready for review October 27, 2021 12:55

sax suggested changes Oct 27, 2021

View reviewed changes

sax reviewed Oct 27, 2021

View reviewed changes

bblaszkow06 reviewed Oct 28, 2021

View reviewed changes

bblaszkow06 self-requested a review October 28, 2021 17:02

mat-hek added a commit that referenced this pull request Nov 2, 2021

efficency & style improvements CR #313

6c83417

andpodob suggested changes Nov 5, 2021

View reviewed changes

andpodob self-requested a review November 8, 2021 10:40

andpodob reviewed Nov 8, 2021

View reviewed changes

bblaszkow06 requested changes Nov 8, 2021

View reviewed changes

mat-hek added 2 commits January 20, 2022 17:33

don't send link requests to elements

c012874

rename functions in the Toilet module

17e9d17

mat-hek force-pushed the auto-demand branch from 42cb5e5 to 17e9d17 Compare January 20, 2022 16:33

mat-hek added 2 commits January 20, 2022 17:36

refactor naming in link handler

4f62cc2

pad props overhaul

3380b5f

mat-hek force-pushed the auto-demand branch from 58f8644 to 3380b5f Compare January 21, 2022 13:45

mat-hek added 3 commits January 21, 2022 17:08

Merge master into auto-demands

6dd4848

Merge branch 'master' into auto-demand

71f6577

update changelog

6489365

mat-hek force-pushed the auto-demand branch from 3aa2fae to 6489365 Compare January 21, 2022 16:26

mat-hek requested review from bblaszkow06 and andpodob January 21, 2022 16:38

bblaszkow06 requested changes Jan 21, 2022

View reviewed changes

improve docs in parent spec

b854f8a

mat-hek requested a review from bblaszkow06 January 24, 2022 09:58

bblaszkow06 requested changes Jan 24, 2022

View reviewed changes

improve docs in parent spec

e06fbad

mat-hek requested a review from bblaszkow06 January 24, 2022 13:25

demand_excess -> target_queue_size

73a1b71

mat-hek force-pushed the auto-demand branch from 40ae2e5 to 73a1b71 Compare January 24, 2022 14:21

bblaszkow06 approved these changes Jan 24, 2022

View reviewed changes

mat-hek requested review from andpodob and removed request for andpodob January 24, 2022 14:31

andpodob approved these changes Jan 24, 2022

View reviewed changes

mat-hek merged commit 5f5dd76 into master Jan 24, 2022

mat-hek deleted the auto-demand branch January 24, 2022 15:41

This was referenced Feb 2, 2022

Add option to suppress "unlinked pads" warning #379

Closed

handle_linking_finished always called on all elements #214

Closed

When adding multiple pads handle_pad_added/3 invokes with all pads already in context #258

Closed

bblaszkow06 mentioned this pull request Feb 14, 2022

Ignored pad options in bin #393

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automatic demands #313

Automatic demands #313

mat-hek commented Jun 22, 2021 •

edited

sax left a comment

sax Oct 27, 2021

sax Oct 27, 2021

sax Oct 27, 2021

sax Oct 27, 2021

sax Oct 27, 2021

mat-hek Nov 9, 2021

bblaszkow06 Oct 28, 2021

bblaszkow06 Nov 8, 2021

bblaszkow06 Oct 28, 2021

bblaszkow06 Oct 28, 2021

bblaszkow06 Oct 28, 2021

andpodob Nov 5, 2021 •

edited

andpodob Nov 8, 2021

bblaszkow06 Nov 8, 2021

bblaszkow06 Nov 8, 2021

mat-hek Nov 9, 2021

bblaszkow06 Dec 22, 2021

bblaszkow06 Nov 8, 2021

mat-hek Nov 9, 2021

bblaszkow06 Dec 22, 2021

bblaszkow06 Nov 8, 2021

mat-hek Nov 9, 2021

bblaszkow06 Nov 8, 2021

mat-hek Nov 9, 2021

bblaszkow06 Dec 22, 2021

bblaszkow06 Nov 8, 2021

bblaszkow06 Nov 8, 2021

bblaszkow06 Jan 21, 2022

bblaszkow06 Jan 21, 2022

bblaszkow06 Jan 24, 2022

mat-hek Jan 24, 2022

andpodob left a comment

	if demand <= div(demand_size, 2) and auto_demands_positive?(pad_ref, state) do
	if demand <= div(demand_size, 2) and auto_demands_positive?(data, state) do

		defp auto_demands_positive?(pad_ref, state) do
		PadModel.get_data!(state, pad_ref, :demand_pads)

	{:stopped, :perpared} -> Core.Child.PadController.check_for_unlinked_static_pads(state)
	{:stopped, :prepared} -> Core.Child.PadController.check_for_unlinked_static_pads(state)

		@spec check_auto_demand(Pad.ref_t(), State.t()) :: State.t()
		def check_auto_demand(pad_ref, demand_decrease \\ 0, state) do

	toilet_size = :atomics.add_get(toilet, 1, buf_size)
	toilet_size = :atomics.add_get(toilet, @toilet_size_idx, buf_size)

	toilet_size = :atomics.add_get(toilet, 1, buf_size)
	toilet_size = AtomicCounter.add_get(toilet, buf_size)

Automatic demands #313

Automatic demands #313

Conversation

mat-hek commented Jun 22, 2021 • edited

The idea

The implementation

Relaying demands

Atomic toilet

Toilet overflow detection algorithm

Bin linking strategy

To do

Further work (issues to be created):

sax left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andpodob Nov 5, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andpodob left a comment

Choose a reason for hiding this comment

mat-hek commented Jun 22, 2021 •

edited

andpodob Nov 5, 2021 •

edited