Proposal: new Database API (Breaking changes please read!)

[davideast]

Just let me use it! - Take the new API for a spin on StackBlitz. Make sure to plug in your own Firebase configuration first.

PR is at #1156

---

Introducing the new AngularFire Database API

The march towards the final release of AngularFire moves on!

We're making improvements to the Database API. Breaking changes are not taken lightly. If we're going to change something, we're going to make it worth your while. Trust me, these new features are well worth it.

This new design keeps all* the same old features, save one, and brings a whole new set of features designed for the modern Angular, RxJS, and ngrx world.

Here's a list of what's coming:

• Generic Service API
• Flexible event streaming
• Simplified Querying API
• ngrx integration
• 55% smaller!

Generic Service API

The current Database API focuses on streaming values as lists or objects. Data operations such as set(), update(), and remove() are available as custom Observable operators.

// current API
constructor(db: AngularFireDatabase) {
    db.list('items').subscribe(console.log);
    db.object('profile/1').subscribe(console.log);
    db.list('items').push({ name: 'new item' }).then(console.log);
}

These data operation methods are not really operators. They return a Promise, not an `Observable. This has caused confusion and several issues when chaining in an observable stream. There is also no type safety in the data operation methods.

The new API removes all custom operators and Observables and provides a new generic service for data operations and streaming.

// NEW API! 🎉
constructor(db: AngularFireDatabase) {
    const itemsList = db.list<Item>('items');
    const items$: Observable<Item[]> = itemList.valueChanges();
    items$.subscribe(console.log);
    itemList.push({ name: 'new item' }); // must adhere to type Item 
}

Instead of returning an FirebaseListObservable from AngularFireDatabase#list(path: string), it now returns a ListReference<T>. This new service is not an Observable itself, it provides methods for creating observables and performing data operations.

Below is the full API reference. If you're curious about the new methods, keep reading!

interface ListReference<T> {
  query: DatabaseQuery;
  valueChanges<T>(events?: ChildEvent[]): Observable<T[]>;
  snapshotChanges(events?: ChildEvent[]): Observable<SnapshotAction[]>;
  stateChanges(events?: ChildEvent[]): Observable<SnapshotAction>;
  auditTrail(events?: ChildEvent[]): Observable<SnapshotAction[]>;
  update(item: FirebaseOperation, data: T): Promise<void>;
  set(item: FirebaseOperation, data: T): Promise<void>;
  push(data: T): firebase.database.ThenableReference;
  remove(item?: FirebaseOperation): Promise<any>;
}

interface ObjectReference<T> {
  query: DatabaseQuery;
  valueChanges<T>(): Observable<T | null>;
  snapshotChanges<T>(): Observable<SnapshotAction>;
  update(data: T): Promise<any>;
  set(data: T): Promise<void>;
  remove(): Promise<any>;
}

Flexible Event Streaming

The current API coalesces all child events into an array. This allows you to easily keep your local data in sync with your database. However, the only option is to listen to all events.

If you want only "child_added" and "child_removed" you'll have to implement your own thing. This is a shame because the FirebaseListObservable does most of this logic.

The new API still coalesces all child events into array, however it allows you to provide a list of events to listen to.

// NEW API! 🎉
constructor(db: AngularFireDatabase) {
    const itemsList = db.list<Item>('items');
    const items$: Observable<Item[]> = itemList.valueChanges(['child_added', 'child_removed');
    items$.subscribe(console.log); // only receive "child_added" and "child_changed" events
}

The events array parameter is optional, by default you will receive all events.

Simplified Query API

Querying with the previous API required too much knowledge of valid query combinations. The new API provides a call back in which you can return a query from.

// current API
constructor(db: AngularFireDatabase) {
   const sizeSubject = new Subject();
   sizeSubject.startWith('large');
    db.list('items', { 
      query: query: {
        orderByChild: 'size',
        equalTo: sizeSubject
      }
    }).subscribe(console.log);
}

This allows you to create any valid query combination using the Firebase SDK. When the Subject emits a new value, the query will automatically re-run with a new set of values. This is an amazing feature, but it can be cumbersome to use.

First of all, the object configuration can't lead to invalid combinations that aren't know until runtime. Secondly, it's a clunkier syntax. Lastly, it hides what's really going on by passing the Subject/Observable to the query.

Rather than require you to know the combinations, use a clunkier syntax, and hide data updates, we can use a simpler API that fits into RxJS conventions.

// NEW API! 🎉
constructor(db: AngularFireDatabase) {
    const size$ = new Subject().startWith('large');
    size$.switchMap(size => {
      return this.db.list('todos', ref => 
        ref.orderByChild('size').equalTo('large'))
        .valueChanges(['child_added', 'child_changed']);
    });
}

Now it's even easier to formulate a dynamic query by just using RxJS operators. The callback allows for much more flexibility than simply changes the criteria values. Within this callback you can change the reference location, ordering method, and criteria.

Now rather than hiding the updates, we can see it all flow in the observable chain.

ngrx integration

The first version of AngularFire for Angular 2 was written in AtScript back in March 9th, 2015. The library was developed in earnest at the end of 2015/beginning of 2016. This also coincided with the rise of ngrx. Unfortunately (even though Jeff, Rob, and I knew each other very well and sat in the same room once a week) we did not work on any integrations together.

Now that I have spent the last year working with ngrx, I wanted to design the library to fit nicely with its conventions.

Action based API

The valueChanges() method returns your data as JSON in either an object or list, but there are others to get your data. You can use snapshotChanges() which returns an array of SnapshotAction. This type acts like a Redux action and preserves the Firebase DatabaseSnapshot and provides other important information like it's event (child_added, etc..) andprevKey.

interface SnapshotAction {
     type: string;
     payload: DatabaseSnapshot;
     key: string;
     prevKey: string | undefined;
}

Build custom data structures with event streaming

Sometimes getting back data as a simple list or object isn't exactly what you need. Rather, you'd like to get back the realtime stream of events and fit them to your own custom data structure. We've introduced two methods to help with that: stateChanges() and auditTrail().

The difference between the two is that auditTrail() waits until the initial data set has loaded (like valueChanges()) before emitting. Using these methods you can consume the stream of events at a location and send it to your reducer and store it as you like.

  ngOnInit() {
    // create the list reference
    this.todoRef = this.db.list('todos');

    // sync state changes with dispatching
    this.todoRef.stateChanges()
      .do(action => {
        this.store.dispatch(action);
      })
      .subscribe();

    // select from store
    this.todos = this.store.select(s => s.todos);
}

This makes implementing reducers really, really, easy. The example below is superfluous because it's just creating an array from child events. This is what snapshotChanges() will give you automatically.

However, you can see that using these state based methods you can formulate whatever state structure you want.

// This is basically what snapshotChanges() does, but it's an example
// of how you can use stateChanges() to formulate a reducer to match your
// desired data structure.
export function todos(state: Todo[] = [], action: SnapshotAction): Todo[] {
  switch (action.type) {
    case TodoActions.ADDED: 
      return [...state, action.payload.val()];
    case TodoActions.CHANGED:
      // re-map todos with new todo
      return state.map(t => t.key === action.key ? action.payload.val() : t);
    case TodoActions.REMOVED:
      return state.filter(t => t.key !== action.key);
    default:
      return state;
  }
}

55% smaller!

From 25.9kb to 11.7kb. When you gzip, it's only 2.7kb!!!

One of the amazing things we were able to do is add all these features, but significantly shrink the library's size. This is due to careful planning and letting RxJS do the hard work. Most of the savings came from removing the old dynamic query API.

Naming

I thought carefully about these names and tried my best to fit them with Angular, Firebase, RxJS and ngrx idioms. However, I'm not really good at naming things. If you find a name confusing or have a better idea please do not hesitate to leave a comment.

database-deprecated

We know this a big change, but we do think it's worth the upgrade. However, we don't want to force you off immediately if you are not ready. We are considering having a angularfire2/database-deprecated module so you can move forward with changes with other modules for a set period of time.

What's next?

If you think this is a great addition, just you wait. We have plans for Angular Universal support and lazy loading of feature modules without the need of the router! We think that these two features together can help improve page load performance.

We are also going to make serious improvements to our documentation. We're going to focus on making it example based and cover common scenarios. If you have any other ideas for documentation we'd love to hear them.

Give it a try!

Take the new API for a spin on StackBlitz. Make sure to plug in your own Firebase configuration first.

[ghost]

I came here to read and seek knowledge, but I found nothing :)

[davideast]

@hanssulo Updated it with the full information!

[cartant]

@davideast Does SnapshotAction contain the actual Firebase DatabaseSnapshot? Won't that be a problem, as actions are supposed to be serializable? It will not be compatible with the Redux DevTools.

[davideast]

@cartant it does contain the Firebase DatabaseSnapshot. However, you don't need to store it or even dispatch it. I keep it around because it contains a lot of great information and you can call .val() to make it serializable.

The type property will be a "child_added", "child_changed", "child_moved", or "child_removed" event, which will be duplicate across data domains. What I imagine most people will do is map over the observable and dispatch that object:

itemList.stateChanges()
    .map(a => ({ type: `ITEM_${a.type.toUpperCase()}`, payload: a.payload.val() })
    .do(a => { this.store.dispatch(a); });

We could provide a way to simplify this, but I don't want to add too much code that ngrx and RxJS already provides.

[ValentinFunk]

Really like the new API! This is pretty much what I always wished the API would look like, just a wrapper to integrate firebase and rxjs 👍 Nice job on the size improvements as well although the API to me is the real reason to upgrade as angularfire already was pretty small compared to what firebase itself brings.

Since a lot of this isn't really angular specific, do you think it would be possible to extract the database part of the library as something like RxjsFirebase so that it can be used with firebase-admin as well? I really love the API and this would be really useful on the server as well.

[benlesh]

Some of the examples above have unnecessary Subjects created... for example:

constructor(db: AngularFireDatabase) {
    const size$ = new Subject().startWith('large');
    size$.mergeMap(size => {
      return this.db.list('todos', ref => 
        ref.orderByChild('size').equalTo('large'))
        .valueChanges(['child_added', 'child_changed']);
    });
}

size$ could just be const size$ = Observable.of('large')

[cartant]

@davideast I think by using actions and putting the actual non-serializable snapshot into an action you are going to end up with support issues, as actions are supposed to be serializable. I think any newcomers to ngrx are not going to be aware of this, will use the AngularFire actions as-is and will then wonder why the Redux DevTools, etc. won't work.

[cartant]

I agree with @kamshak on this:

Since a lot of this isn't really angular specific, do you think it would be possible to extract the database part of the library as something like RxjsFirebase so that it can be used with firebase-admin as well?

[davideast]

@benlesh Totally. This is for an example only. This makes it easier to trigger local events when testing around.

@kamshak @cartant I'm totally open to this. I think we'd have to figure out if we'd want to implement the action based APIs for the RxJS Firebase library.

@cartant I see your point about people doing this incorrectly. We could unwrap the snapshot as the payload and keep any needed information on the action. We'd then need to re-think the name snapshotChanges(). But I am open to it.

[Toxicable]

I'm also for a framework agnostic implementation, then provide a simple Angular module which does the injection seprate from the core parts which can be used anywhere.
While we're at it I think now might be a good time to revisit the Lib name?
If we renamed it then we wouldn't have to change the name of the deprecated database module, and in turn not breaking anyones code

[cartant]

@davideast I think the simplest thing to do would be to just change the name. Perhaps something like SnapshotChange rather than SnapshotAction? And maybe rename payload to snapshot so that it looks less actiony? And maybe rename type to event? It is, after all, a Firebase event name and snapshotChanges would be a relatively low-level and close-to-Firebase API.

[cartant]

@davideast I noticed that in your example above you've used mergeMap. The AngularFire observables don't complete; they re-emit a value when the database changes. That means you pretty much always want to use switchMap instead. If mergeMap is used, values from no-longer-relevant observables will be emitted into the stream when the database changes.

[wbhob]

I think the package should be called @angular/firebase for simplicity and to line up with other Angular packages, like material

[cartant]

@davideast Good to see that unwrapMapFn is deprecated/gone. 🎉 👍

[Martin-Andersen]

Is the new API in RC2?

[Toxicable]

No this is just a proposal right now

[ghost]

Changes are looking great! As I am unfamiliar with the entire process of professional software development, is there a time window when this is going to be released approximately?

[willgm]

I believe there is an error at the new query API example: the .equalTo method should receive the parameter size of the switchMap, right? I'm just asking to make sure I understood the proposal.

constructor(db: AngularFireDatabase) {
    const size$ = new Subject().startWith('large');
    size$.switchMap(size => {
      return this.db.list('todos', ref => 
        ref.orderByChild('size').equalTo(size)) // <<~~~~ here, and also an extra parentheses 
        .valueChanges(['child_added', 'child_changed']);
    });
}

[jaufgang]

@Toxicable, @Martin-Andersen, while it does say "Proposal" in the issue title, take note that doesn't mean that these changes are just in an early proposal/design stage. These changes are fully implemented and @davideast posted this issue as a "Proposal" in order to solicit feedback on the new API before releasing it.

If you look closely at the original post, you will see that there is a PR (#1156) in place with for this new API, and a stackblitz sample project which consumes this new API by using angularfire2@4.0.0-rc.3-exp.7

So it looks like these changes are on track to potentially be part of RC3.

That having been said, I feel compelled to point out that using proper semantic versioning, a major breaking API change like this should not be introduced between minor RC releases. The major version number should be bumped up to 5 for this change.

[jaufgang]

I have taken the suggestion by @kamshak to "extract the database part of the library as something like RxjsFirebase" and posted it as a separate issue (#1162) where it can take on a life of it's own and be discussed independently of these particular API changes.

[davideast]

@cartant we can have our cake and eat it too! A DatabaseSnapshot is serializable because it's .toJSON() method returns the result of .val().

[cartant]

@davideast Perhaps I should have said that the action payloads should be already serialized, rather than serializable. The fact that the snapshot is serializable won't help, as actions are serialized when sent to the DevTools (which run in a different page). That means that when time travel debugging, replayed actions will contain JSON and not a snapshot instance, so there will be no val() to call and it will break.

[cartant]

@davideast Just to reiterate, my only concern is with the name. Calling them actions seems like an invitation to misuse them. The implementation looks great.

I just don't see the upside of naming them and structuring them like Redux actions when they are not intended to be dispatched and have payloads that will break if used with the DevTools.

[hiepxanh]

Question 1: do anyone know when is this proposal intend to release ?
Question 2: is that work with angular-redux/store too ??
I need time to prepare to learn redux.

[davideast]

@hiepxanh

1. Today or tomorrow!
2. It works with any redux based system, but it is not required. I recommend ngrx/store

[shakhrillo]

const itemsList = this.db.list('all_posts_id');
const items$: Observable = itemsList.valueChanges(['child_added', 'child_removed']);
items$.subscribe(console.log); // only receive "child_added" and "child_changed" events

@davideast Is there any ways to return only new added element without "child_changed" events?

[nikhil-mahirrao]

list.valueChanges() is not giving keys and list.snapshotChanges() gives too much completex object.
Is there anything which gives array of objects with keys?

[sparqueur]

I agree with nikhil-mahirrao.
In my case I want to do client side filtering for performance purpose.
Therefore I do something like :

classroomsAf: AngularFireList<Classroom>;
classroomsObs: Observable<Classroom[]>;
classroomsValues: Classroom[];
classroomsFiltered: Classroom[];
queryText: string;

ngOnInit() {
    this.classroomsAf = this.afDatabase.list('/classrooms');
    this.classroomsObs = this.classroomsAf.valueChanges();
    this.classroomsObs.subscribe( (classroomValues) => {
      this.classroomsValues = classroomValues;
      this.filterClassrooms();
    });
  }

filterClassrooms() {
    this.classroomsFiltered= _.filter(this.classroomsValues, (classroom) => 
   classroom.name.toLowerCase().indexOf(this.queryText.toLowerCase()) != -1);
}

Problem : I can't delete an item as I "lose" the $key.

Using snapshot is possible but it makes this more complicated for nothing.

Wouldn't it be possible to have an implicite $key mapping on the retrieved objects ?

At least please change SnapshotAction to a generic class in order to have things cleaner 👍

classroomsAf: AngularFireList<Classroom>;
classroomsSnap: Observable<SnapshotAction<Classroom>[]>;
classroomsValues: SnapshotAction<Classroom>[];
classrooms: SnapshotAction<Classroom>[];

Or maybe I doing this wrong ?

[MM3y3r]

I have got a question. I cannot seem to add a key to the object i want to push.. I know i have to use update(). What exactly constitutes a Firebaseoperation?