Enhancement: marbles syntax + testing functions #299

jcafhe · 2019-01-29T09:59:04Z

This PR is for updating the marbles syntax, and providing some helper functions (cold & hot) that mimic the rxjs functions. Some tests and examples have also been added.

Sorry for the following wall of text 😕

Marbles syntax

In the following, "ts" stands for timespan. E.g. 3ts = 3 * timespan.

breaking changes

each character advance time by 1ts (except for space), e.g. -ab-c-- will produce ab at 1ts, c at 4ts instead of ab at 1ts, c at 3ts.
change error symbol from x or X to #
move from_marbles to rx (implemented in rx/core/observable/marbles.py). Alias cold.
move to_marbles to rx.operators (implemented in rx/core/operators/tomarbles.py)

new features

add rx.hot (implemented in rx/core/observable/marbles.py)
add support for grouping marbles e.g. --(ab,c)-d-. Every marble in a group share the same timestamp defined by the first (. Each character (even parentheses) advance time by 1ts. ab and c will be emitted at 2ts, d at 9ts.
add optional lookup dictionnary to convert a marble to the specified value. E.g. from_marbles("a-b--", lookup={'a':11, 'b':22} will produce 11 at 0 and 22 at 2 * timespan.
add optional error parameter to specify the exception raised by # marble.

New functions

`marbles_testing(timespan)`

Python context that setup a TestScheduler and returns the functions below as a namedtuple. This ensures that everything will be called with the same values for the following parameters: subscribed, created, disposed, timespan. Parameters are set to:

created = 100.0
subscribed = 200.0
disposed = 1000.0

Regarding hot() function, if a marble is specified as the first character, it will be skipped.

`cold(string, [lookup, error]) -> Observable`

Parse a marbles string and return a cold observable by calling rx.cold.

`hot(string, [lookup, error]) -> Observable`

Parse a marbles string and return a hot observable by calling rx.hot.

`start(create | observable) -> List[Recorded]`

Wrapper around TestScheduler.start(). Returns a list of records instead of a MockObserver (i.e. mockobserver.messages).

`exp(string, [lookup, error]) -> List[Recorded]`

Parse a marbles string and return a list of records.

coveralls · 2019-01-29T10:02:08Z

Coverage increased (+0.8%) to 87.34% when pulling 8fbe13a on jcafhe:enhancement/marbles into bbc9fec on ReactiveX:master.

rx/testing/marbles.py

dbrattli · 2019-01-29T12:19:55Z

Really nice work! Questions:

How to I emit 42 if 42 means 4 timespan 2 and (42) means 4 and 2. Do I need to use a variable such as a and then a lookup dict(a=42)?
There is also a mix of schedulers, and I think its nice that we can set schedulers so use this in real code and not only for testing. However calls like create_cold_observable is only available for the TestScheduler.
Because of 2, we should consider moving this out of testing and into operators/observable. What do you thing?

jcafhe · 2019-01-29T13:22:27Z

How to I emit 42 if 42 means 4 timespan 2 and (42) means 4 and 2. Do I need to use a variable such as a and then a lookup dict(a=42)?

Yes, this is the big drawback of one character = one timespan, you have to use the lookup dict if your value cannot be represented by only one character. In the contrary, it makes possible to emit consecutive values. Otherwise you would have to write "-4-2-", and divide by two your timespan. Do you think we lose too much usability with this change?

There is also a mix of schedulers, and I think its nice that we can set schedulers so use this in real code and not only for testing. However calls like create_cold_observable is only available for the TestScheduler.

You're right, test_context() function returns hot() and cold() bound to the test scheduler.

In a normal/test context, we could just alias from_marbles() to cold() since from_marbles still uses your implementation, I've just extract the parsing part so it is not bound to the test scheduler. In fact, even in a test context, we don't really need testscheduler.create_cold_observable(), just need to use from_marbles.

Regarding hot(), it's unclear for me. How would we specify when the emission should start ? Maybe just by giving an absolute time? Do you think we can implement this with a subject ?

Because of 2, we should consider moving this out of testing and into operators/observable. What do you thing?

Yes, from_marbles is much more that testing, so it makes sense. Concerning to_marbles() I'm not convinced (seems to be not reliable).

…) function + typo

jcafhe · 2019-01-29T18:49:40Z

I've come to a solution for creating a hot observable with publish & delay operators. It should work with any schedulers. Need a bit more time to test it.

The basic idea is to provide a scheduler to hot() as well as a datetime/timedelta to determine when to start emitting marbles.

I think we should also get rid of the subscription symbol '^'. It complicates things and I don't see what the purpose of it, even in tests. Don't know why they have introduced it in rxjs. What do you think ?

MainRo · 2019-01-29T20:57:54Z

change error symbol from x or X to #

Yes, if alpha numerical values are allowed as item values (i.e. [a-zA-Z0-9]), then we must use special characters for signaling.

Yes, this is the big drawback of one character = one timespan, you have to use the lookup dict if your value cannot be represented by only one character. In the contrary, it makes possible to emit consecutive values. Otherwise you would have to write "-4-2-", and divide by two your timespan. Do you think we lose too much usability with this change?

We could separate groups by timespans (-) and still consider each digit as a timespan. For example "-42-3-" would mean that 42 is emitted at 1ts and 3 at 4ts. I even find this more readable because each emitted item is clearly separated from the previous/next ones. Also such a syntax could be reusable for ascii-to-marble docs because timespans would be consistent on multiple lines. For example something like this:

---1--9---3---|

[ map(x: x*2) ]

---2--18--6---|

is quite readable on ascii, and timelines alignment is preserved on both observables even though they emit items of different "length". This is the current syntax that I consider for documentation now.

Such a syntax has pros for documentation purpose, but may not be applicable for testing and serialization: On the previous example, an additional timespan is needed after the 9 value to keep text alignment with the second observable. So at the end I am not sure that we can use the very same syntax for doc and serialization/tests.

I think we should also get rid of the subscription symbol '^'. It complicates things and I don't see what the purpose of it, even in tests. Don't know why they have introduced it in rxjs. What do you think ?

I agree. Also I do not see when cold vs hot observable difference is needed here (especially if we remove the subscription symbol).

rx/testing/marbles.py

dbrattli · 2019-01-30T07:28:00Z

Agree to remove subscription symbol. It's important that we don't add features before we need them.

I think the map times 2 example to @MainRo is very interresting since we easily get into the problem of how to represent multi-digit numbers, and I think it would be strange if to_marbles() could not represent them. I like @MainRo suggestion about having each digit represent a timespan. For to_marbles there is then the possibility of overlapping values with larger numbers, but that could perhaps be fixed by wrapping the output if that should happen e.g:

---1-2-------|

[ map(x: x*1000) ]

---1000
     2000----|

Q: Is the plan to fix to_marbles() as part of this PR, or will that be a separate PR later?

jcafhe · 2019-01-30T09:02:22Z

Agree with @MainRo too, it should be a good compromise. There's still a case we need to clarify:
how to deal with values in a group ? I'd suggest to add a comma, to separate values:

--(12,4,5)--6--
01234567890123

So 12, 4, 5 will be pushed at 2ts, and 6 at 12ts.

Q: Is the plan to fix to_marbles() as part of this PR, or will that be a separate PR later?

@dbrattli no, it was not planned. It should be better to make a later PR since the current PR may be too heavy.

I was thinking about moving from_marbles alias cold as well as hot in an implementation file rx/core/observable/marbles.py, then reference cold& hot in rx.init_.py.
In parrallel, we keep in rx/testing/marbles (or even in rx/testing/reactivetest) a true python context that returns a cold& hot working with the test scheduler:

with context as (start, cold, hot, exp):
    e0 = cold("--a--b--|")
    ex = exp(" --a--b--|")
    results = start(e0)
    assert results == ex

Of course, cold and hot in this test context would rely on rx/core/observable/marbles.py and not on testscheduler.create_cold/hot_observable.

…rbles to rx.core.observable.marbles + update tests & examples

jcafhe · 2019-01-30T17:59:02Z

@dbrattli @MainRo What should we do with negative numbers ? Negative sign conflits with our time progression symbol.

Maybe we can choose an other symbol for time progression e.g. underscore or tilde ?

____ab__c_125_|
___-6__c__12.5__1e-5___#

~~~ab~~c~125~|
~~~-6~~c~~12.5~~1e-5~~#

Or just restricting to positive numbers and forcing the use of the lookup dict for negative numbers.

dbrattli · 2019-01-30T19:44:07Z

Stick to using -. If we need support for negative number we should "escape" them with parenthesis e.g:

---(-6)--c--12.5--(1e-5)---#

MainRo · 2019-01-30T21:03:48Z

This may be another reason why cold/testing syntax may not be exactly the same than for documentation: For documentation purpose there is no need to encode any possible values, this is why I proposed to restrict values to "[a-zA-Z0-9]". For cold/testing we probably want to represent more values but it will still be somehow limited (no objects, no bytes...).

dbrattli · 2019-01-31T09:08:14Z

examples/marbles/frommarbles_merge.py

@@ -13,3 +15,5 @@
    on_completed=lambda: print('good job!'),
    scheduler=ccy.timeout_scheduler,
    )
+


Have you considered using .run() instead? It's made for exactly this scenario. It will then return the last value emitted, and you can pipe to ops.to_iterable() to catch them all.

Yes you're right, good idea !

…ests

…observable/test_marbles.py

dbrattli · 2019-02-03T15:49:05Z

examples/marbles/frommarbles_flatmap.py

+    ops.flat_map(lambda value: observableLookup[value]),
+    )
+
+source.subscribe_(


Use subscribe() from examples now that subscribe supports (again) both observers and callbacks. Subscribe with an _ will still be used by the operators, but it may be confusing for the users that we have two different functions.

Noted 👍
@dbrattli By the way, I've reworked all examples + some others things waiting in a big commit. I'm sorry this PR is a bit of a mess because I've pushed a lot of commits that changes everything. Do you prefer I close this one and open a new one when it's ready?

@jcafhe I think it's time to squash and merge. This feature is self contained and I'm not worried if it needs more work.

make marble context a true python context update tests update examples

jcafhe · 2019-02-03T18:08:27Z

Alright,

sorry for the multiple commits, I've had to rearange a lot of things (and made some mistakes).
Now, things are a bit more clear:

from_marbles is now part of the rx API and implemented in rx/core/observable/marbles.py as well as hot
to_marbles is now exposed in rx.operators API and implemented in rx/core/operators/tomarbles.py
the marbles syntax we discussed above should be supported now
the context marbles_testing is available in rx.testing.marbles and need to be used inside a with statement
tests relative to rx.from_marbles and rx.hot have been moved to tests/test_observable/test_marbles.py
tests relative to the marbles_testing context are still in tests/test_testing/marbles.py
use of run instead of time.sleep() in examples

lock · 2020-02-08T00:54:55Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

Jérémie Fache added 2 commits January 29, 2019 10:33

complete marbles syntax + add tests + add new testing functions

367e75c

sleep the main thread in examples

ae22327

dbrattli reviewed Jan 29, 2019

View reviewed changes

rx/testing/marbles.py Outdated Show resolved Hide resolved

dbrattli reviewed Jan 29, 2019

View reviewed changes

rx/testing/marbles.py Outdated Show resolved Hide resolved

dbrattli reviewed Jan 29, 2019

View reviewed changes

rx/testing/marbles.py Outdated Show resolved Hide resolved

jcafhe added 2 commits January 29, 2019 14:37

fix the scheduler selection mechanism

dbe6856

remove dependency to testscheduler.create_cold_observable() for cold(…

0778533

…) function + typo

implement hot not dependent to test scheduler

7f918ee

dbrattli reviewed Jan 29, 2019

View reviewed changes

rx/testing/marbles.py Outdated Show resolved Hide resolved

move from_marbles(), to_iterable(), hot(), parse() from rx.testing.ma…

812aba3

…rbles to rx.core.observable.marbles + update tests & examples

remove dependency to rx.testing.recorded.Recorded() + update test

b7b6dfe

dbrattli reviewed Jan 31, 2019

View reviewed changes

Jérémie Fache added 8 commits January 31, 2019 19:14

fix exp() function

39d9131

add to_marbles in rx.__init__.py

3f5224b

implement the marbles syntax as discussed in ReactiveX#299 + update t…

1efbea6

…ests

add tests for rx.hot

bc5cef0

implement hot with no operators

3c6c97c

move tests relative to from_marbles()/cold() and hot() to tests/test_…

aaaeeb1

…observable/test_marbles.py

fix elements should be skipped after on_completed or on_error

b774bfd

handle timedelta/datetime + enforce time value as float + cleaning

c44e948

dbrattli reviewed Feb 3, 2019

View reviewed changes

move to_iterable in rx/operators instead of rx/observable

1df1055

make marble context a true python context update tests update examples

typo

8fbe13a

dbrattli approved these changes Feb 5, 2019

View reviewed changes

dbrattli merged commit 379b699 into ReactiveX:master Feb 6, 2019

jcafhe deleted the enhancement/marbles branch February 7, 2019 17:00

lock bot locked as resolved and limited conversation to collaborators Feb 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhancement: marbles syntax + testing functions #299

Enhancement: marbles syntax + testing functions #299

jcafhe commented Jan 29, 2019 •

edited

Loading

coveralls commented Jan 29, 2019 •

edited

Loading

dbrattli commented Jan 29, 2019

jcafhe commented Jan 29, 2019 •

edited

Loading

jcafhe commented Jan 29, 2019 •

edited

Loading

MainRo commented Jan 29, 2019

dbrattli commented Jan 30, 2019

jcafhe commented Jan 30, 2019

jcafhe commented Jan 30, 2019 •

edited

Loading

dbrattli commented Jan 30, 2019

MainRo commented Jan 30, 2019

dbrattli Jan 31, 2019

jcafhe Jan 31, 2019

dbrattli Feb 3, 2019

jcafhe Feb 3, 2019 •

edited

Loading

dbrattli Feb 4, 2019

jcafhe Feb 4, 2019

jcafhe commented Feb 3, 2019 •

edited

Loading

lock bot commented Feb 8, 2020

Enhancement: marbles syntax + testing functions #299

Enhancement: marbles syntax + testing functions #299

Conversation

jcafhe commented Jan 29, 2019 • edited Loading

Marbles syntax

breaking changes

new features

New functions

marbles_testing(timespan)

cold(string, [lookup, error]) -> Observable

hot(string, [lookup, error]) -> Observable

start(create | observable) -> List[Recorded]

exp(string, [lookup, error]) -> List[Recorded]

coveralls commented Jan 29, 2019 • edited Loading

dbrattli commented Jan 29, 2019

jcafhe commented Jan 29, 2019 • edited Loading

jcafhe commented Jan 29, 2019 • edited Loading

MainRo commented Jan 29, 2019

dbrattli commented Jan 30, 2019

jcafhe commented Jan 30, 2019

jcafhe commented Jan 30, 2019 • edited Loading

dbrattli commented Jan 30, 2019

MainRo commented Jan 30, 2019

dbrattli Jan 31, 2019

Choose a reason for hiding this comment

jcafhe Jan 31, 2019

Choose a reason for hiding this comment

dbrattli Feb 3, 2019

Choose a reason for hiding this comment

jcafhe Feb 3, 2019 • edited Loading

Choose a reason for hiding this comment

dbrattli Feb 4, 2019

Choose a reason for hiding this comment

jcafhe Feb 4, 2019

Choose a reason for hiding this comment

jcafhe commented Feb 3, 2019 • edited Loading

lock bot commented Feb 8, 2020

jcafhe commented Jan 29, 2019 •

edited

Loading

`marbles_testing(timespan)`

`cold(string, [lookup, error]) -> Observable`

`hot(string, [lookup, error]) -> Observable`

`start(create | observable) -> List[Recorded]`

`exp(string, [lookup, error]) -> List[Recorded]`

coveralls commented Jan 29, 2019 •

edited

Loading

jcafhe commented Jan 29, 2019 •

edited

Loading

jcafhe commented Jan 29, 2019 •

edited

Loading

jcafhe commented Jan 30, 2019 •

edited

Loading

jcafhe Feb 3, 2019 •

edited

Loading

jcafhe commented Feb 3, 2019 •

edited

Loading