New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Help Wanted: How to flatmap multiple queries together #102

Closed
digitalbuddha opened this Issue Mar 23, 2016 · 6 comments

Comments

4 participants
@digitalbuddha

digitalbuddha commented Mar 23, 2016

My app has a data model of a collection of routes that each contain a collection of stops which each contain a collection of payload items. I created a query to load routes, one to load events for a route and one to load payloads for an event. I then wrapped them in RxJava operators to flatmap together in a similar fashion as if this was network data.

 public Observable<List<RouteSQL>> loadRoutes() {
        return db.createQuery(TABLE_NAME, SELECT_ALL)
                .mapToList(MAPPER::map)
                .first()
                .flatMap(Observable::from)
                .flatMap(this::withEvents)
                .cast(RouteSQL.class)
                .toList();
    }

 public Observable<ImmutableRouteSQL> withEvents(RouteSQL route) {
        return loadEvents(route.id())
                .map(((ImmutableRouteSQL) route)::withTransit_events);
    }

    @NonNull
    public Observable<List<TransitEventSQL>> loadEvents(Long routeId) {
        return db.createQuery(TransitEventSQL.TABLE_NAME,
                TransitEventSQL.SELECT_FOR_ROUTE,
                new String[]{String.valueOf(routeId)})
                .mapToList(TransitEventSQL.MAPPER::map)
                .first()
                .flatMap(Observable::from)
                .flatMap(this::eventWithPayload)
                .toList();
    }

 private Observable<ImmutableTransitEventSQL> eventWithPayload(TransitEventSQL event) {
        return loadPayloads(event.id())
                .map(((ImmutableTransitEventSQL) event)::withTransit_event_payloads);
    }

    @NonNull
    private Observable<List<Payload>> loadPayloads(long eventId) {
        return db.createQuery(Payload.TABLE_NAME,
                Payload.SELECT_BY_TRANSIT_ID,
                new String[]{String.valueOf(eventId)})
                .mapToList(Payload.MAPPER::map).first();
    }

The odd piece of code up top is all the first() sprinkled after each mapTo. Without them it seems like I cannot take query for a list, flatmap the list to a stream, then map each entry to another query result and reduce back to a list. The "issue" lies in each query being an endless observable. Am I missing something or is this the proper way to join multiple result sets? Ideally I'd like to be able to flatmap multiple sql brite queries together without having to do the first trick. Thank you for help.

@phajduk

This comment has been minimized.

Show comment
Hide comment
@phajduk

phajduk Mar 23, 2016

Contributor

As I understand, QueryObservable is some kind of endless observable. It will emit value to subscribers when supplied table's data change (in the future). You probably expect similar behavior as if this was network data, but in network you have call->response finished by onCompleted/onError without longer lifecycle. However I would wait for maintainers' opinion.

Contributor

phajduk commented Mar 23, 2016

As I understand, QueryObservable is some kind of endless observable. It will emit value to subscribers when supplied table's data change (in the future). You probably expect similar behavior as if this was network data, but in network you have call->response finished by onCompleted/onError without longer lifecycle. However I would wait for maintainers' opinion.

@JakeWharton

This comment has been minimized.

Show comment
Hide comment
@JakeWharton

JakeWharton Mar 30, 2016

Collaborator

You definitely shouldn't need the first() calls. The problem here is that you need to nest your observables inside the flatMap instead of trying to flatten them to a single stream.

Something like:

routesObservable
    .mapToList(..)
    .flatMap(list -> {
        return Obserable.from(list)
            .flatMap(this:withEvents)
            .toList();
    });

with the same nesting behavior inside withEvents. This maintains each List being emitted at the root while still mixing in additional properties.

Now, that said, I would urge you to think about how queries are done differently. While the "routes" have multiple "stops", don't try to structure these as a parent/child relationships like you would in an ORM.

If you are displaying a list of routes and each route says the number of stops it has you should only be querying for routes and included the count of stops in that query. Then, if the user clicks on a route to display a single one, you would query for that single route as one query and then use a separate query for the route's stops (which might include the count or summary of its payload items). This ensures that only the data being updated triggers a new query for that data. An update to the route only re-triggers the route query and you UI only updates the route info. A new stop on that route only re-triggers the list of stops query to re-trigger and the list adapter to update.

Collaborator

JakeWharton commented Mar 30, 2016

You definitely shouldn't need the first() calls. The problem here is that you need to nest your observables inside the flatMap instead of trying to flatten them to a single stream.

Something like:

routesObservable
    .mapToList(..)
    .flatMap(list -> {
        return Obserable.from(list)
            .flatMap(this:withEvents)
            .toList();
    });

with the same nesting behavior inside withEvents. This maintains each List being emitted at the root while still mixing in additional properties.

Now, that said, I would urge you to think about how queries are done differently. While the "routes" have multiple "stops", don't try to structure these as a parent/child relationships like you would in an ORM.

If you are displaying a list of routes and each route says the number of stops it has you should only be querying for routes and included the count of stops in that query. Then, if the user clicks on a route to display a single one, you would query for that single route as one query and then use a separate query for the route's stops (which might include the count or summary of its payload items). This ensures that only the data being updated triggers a new query for that data. An update to the route only re-triggers the route query and you UI only updates the route info. A new stop on that route only re-triggers the list of stops query to re-trigger and the list adapter to update.

@JakeWharton

This comment has been minimized.

Show comment
Hide comment
@JakeWharton

JakeWharton Mar 30, 2016

Collaborator

Actually the nested example I gave won't work. An update to the "withEvents" observable will cause duplicates in the list. In that case, you're better off actually adding the Observable of events, not the actual list of events, to the route for downstream code to subscribe.

Again, though, it's best to try and avoid the ORM-like object construction for parent/child relationships.

Collaborator

JakeWharton commented Mar 30, 2016

Actually the nested example I gave won't work. An update to the "withEvents" observable will cause duplicates in the list. In that case, you're better off actually adding the Observable of events, not the actual list of events, to the route for downstream code to subscribe.

Again, though, it's best to try and avoid the ORM-like object construction for parent/child relationships.

@digitalbuddha

This comment has been minimized.

Show comment
Hide comment
@digitalbuddha

digitalbuddha Mar 30, 2016

Thanks Jake. I completely missed that pattern of holding observables rather
than lists within my models. Normally there's no reason to do the Orm
mapping, this is for the use case of needing to sync full data model back
to server.
On Mar 30, 2016 1:46 AM, "Jake Wharton" notifications@github.com wrote:

Actually the nested example I gave won't work. An update to the
"withEvents" observable will cause duplicates in the list. In that case,
you're better off actually adding the Observable of events, not the
actual list of events, to the route for downstream code to subscribe.

Again, though, it's best to try and avoid the ORM-like object construction
for parent/child relationships.


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#102 (comment)

digitalbuddha commented Mar 30, 2016

Thanks Jake. I completely missed that pattern of holding observables rather
than lists within my models. Normally there's no reason to do the Orm
mapping, this is for the use case of needing to sync full data model back
to server.
On Mar 30, 2016 1:46 AM, "Jake Wharton" notifications@github.com wrote:

Actually the nested example I gave won't work. An update to the
"withEvents" observable will cause duplicates in the list. In that case,
you're better off actually adding the Observable of events, not the
actual list of events, to the route for downstream code to subscribe.

Again, though, it's best to try and avoid the ORM-like object construction
for parent/child relationships.


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#102 (comment)

@JakeWharton

This comment has been minimized.

Show comment
Hide comment
@JakeWharton

JakeWharton Aug 20, 2016

Collaborator

Closing since this is fairly old. Let me know if this hasn't been fully resolved.

Collaborator

JakeWharton commented Aug 20, 2016

Closing since this is fairly old. Let me know if this hasn't been fully resolved.

@kmfish

This comment has been minimized.

Show comment
Hide comment
@kmfish

kmfish Dec 21, 2016

In our project, we also flatmap multiple QueryObservable like @digitalbuddha . We also found the problem that "An update to the "withEvents" observable will cause duplicates in the list." Our workaround is, use a QueryObservableManager object for one Observable subcribe chain. It maintains all subscriptions of QueryObservable in the subscribe chain. It will unsubcribe existed subscription except last one for a special queryObservable.
QueryObservableManager implement

kmfish commented Dec 21, 2016

In our project, we also flatmap multiple QueryObservable like @digitalbuddha . We also found the problem that "An update to the "withEvents" observable will cause duplicates in the list." Our workaround is, use a QueryObservableManager object for one Observable subcribe chain. It maintains all subscriptions of QueryObservable in the subscribe chain. It will unsubcribe existed subscription except last one for a special queryObservable.
QueryObservableManager implement

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment