EmberJS model doesn't detect deleted records from the DB #962

Closed
ddewaele opened this Issue May 5, 2013 · 28 comments

Comments

Projects
None yet

ddewaele commented May 5, 2013

I have a very simple CRUD app (https://github.com/ddewaele/emberjs-crud-rest) using EmberData, the REST adapter, and a Ember.JS compatible REST backend that exhibits the following problem :

When records are deleted from the DB outside of the EmberApp (ex: through the DB console) , EmberJS never detects that the record or records are deleted.

A call to App.WhateverModel.find() keeps returns these deleted records despite the fact that I see the find() call going to the REST backend.

So it should know that these records got deleted.

As the find() call is capable of detecting new records (App.WhateverModel.find() sees records that were inserted in the DB outside of the Ember.JS app), I wonder if this is a bug.

If it's not a bug, it's not clear to me howto "reset" the model so that it picks up the new data.

The only workaround right now is to do a full page refresh using the browser F5.

Member

tchak commented May 5, 2013

Yes this is a very real problem. There is a discussion going on in here #379. First thing we need is define a semantics to side delete records. For now a work around is for you to find a way to notify your client about deleted ids and manually call unloadRecord on concerned records.

ddewaele commented May 5, 2013

OK. Like with many things Ember I'm never sure if it's a bug or a complete misunderstanding on my part :)
Thanks for taking the time to respond so quickly....

Would be nice if there is a published workaround for this. I mean, unless I'm mistaken this is a very common use-case. This issue might not popup in single-user demo apps but any app that does some kind of CRUD using EmberData is affected by this.

Unfortunately I'm not a javascript / EmberJS guru so I have no idea how to notify my clients about deleted ids.

ddewaele commented May 5, 2013

@tchak Went through issue #379 but don't understand why the server needs to send meta-data for this. Don't want to add myself to the discussion as I'm hardly the most knowledgable javascript / emberJS person around.

But most backend servers will not be bothered to send meta-data regarding deleted records I guess. It seems bizarre that the backend service needs to keep track of what's deleted (even in light of jsonapi.org).

Coming from an enterprise background I would just expect that EmberData has the capability to return me whatever is in the store, regardless of what I have locally. Or at least provide some kind of global reset or refresh method, or mark the object as being deleted server-side.

The thing is that I see EmberData going to my REST backend service and it is retrieving all the up-to-date data. So why can't my EmberJS app make use of that.

I wonder how people are currently dealing with this issue if they have more than 1 user. Doesn't this mean that there is always somebody who is going to see state data (If user 1 deletes something in the DB user 2 will never see that it got deleted ?)

Member

tchak commented May 5, 2013

@ddewaele I agree with you that it can be quite common. Unfortunately, there is no simple api/backend semantics that everyone agrees on as of today.
I think, that lots of people deal with it by using only explicit has many relations and by updating the parent. I am not saying it is a good solution, but it is a one that could work...

Member

tchak commented May 5, 2013

@ddewaele Well the server have to notify your client at some point. The client can not just guess :) If you want to get automatic notification about deleted records, you will have to go through some kind of push system (websocket for example). In this case your server do not have to keep track of deleted records.

In case you call find with an id of already loaded record, Ember Data will use cached version and will newer try to reload the record by calling find() on the adapter. So the client have no way to know the record was deleted.
What we could do, is in case of call to reload() and if getting a 404 error we could assume te record should be unloaded. Error handeling in Ember Data is not optimal for now, but we hope to get it better very soon.

ddewaele commented May 5, 2013

Even more confused now :)

I have this issue with a single entity datamodel
No relations, no parents, .... just 1 model - 1 type of record.
Somebody deletes a record in the DB.
I want all my users accessing the app to see that.
I expected a call to find() (or another call) to be able to deliver that.

I don't get why you need api / backend semantics for that. I think most people can add some flexibility into their API design to accomodate something like jsonapi.org , but adding side-deleting semantics as proposed in #379 might be stretching it. But then again, I might be missing the point completely.

If somebody has a simple workaround for me you can always post it on SO :
http://stackoverflow.com/questions/16380143/emberjs-model-find-not-up-to-date-with-underlying-store

But it's time to look at some alternatives I'm afraid.....

I hope you'll find a solution for this.

Member

tchak commented May 5, 2013

@ddewaele the problem is you expect a call to findAll to reset your store. This is not how Ember Data works. You have to think of Ember Data store as if it was an in memory DB on it own. So you have to "sync" it with your backend. Simply pull of your new data from the backend will only update existing records and add missing ones, but there is no way to know what should be deleted. Sorry if I am not very clear in my explications :)

ddewaele commented May 5, 2013

@tchak Just to clarify , I was not expecting to see a real-time update. I agree that this would imply some kind of server -> client push. But this is not a requirement. It's much simpler than that. The user simply needs to be able to perform an action in the app to "refresh" the data (clicking a refresh button, or navigating back to an overview screen).

Coming back to my use-case: "Users goes to the overview screen and sees up-to-date data from the backend"

I guess I'm looking for a simple way to "reset" the store. However, I didn't find such a thing in EmberData. For me it make sense to have a use-case where the user sees the latest data from the backend, regardless of what it has locally. Is such a "reset" possible ? Performance-wise there's no difference.
"The way Ember Data works now" also requires a call to the backend effectively retrieving all data.

Regarding "how Ember Data works" :

Even if EmberData is an in-memory DB on its own, it was at some point synced with the backend the first time I called find(). I assumed that EmberData would have enough information it needs to detect that certain records were deleted.

-It knows what records it has stored locally (in memory store) as a result of the first find() call.
-It knows what records come back from the REST backend.

If there are records in the in-memory store that aren't dirty and they didn't come back from the REST backend then you could assume they are deleted and they should be removed from the in-memory-db too no ?
What's the added-value of ignoring these deleting records complete and still returning them as if they were still present in the backend ?

It seems that the developer will need to do a lot of plumbing to implement such a "sync" himself.

Member

tchak commented May 5, 2013

@ddewaele what you propose is some kind of invalidation based on the url that loaded your records. You also assume that /resources returns all of the records available on the backend. Even if it was true, there no one and only way to load "all" records. Actually, Ember Data will newer fire two times the same /resource request. It will call it only once and then you have two options : force it by calling find({}) in wich case you will fire an empty query, and you can newer assume identity of a query, or by calling update() on the records collection, in which case you will trigger a call to /resources url but with attached to it since token, so the result is supposed to contain only modified records. This is where came in the discussion about side-deleting. A server should be able to return a list of deleted records since last request. Trying to make a diff on the client seems dangerous and error prone.

ddewaele commented May 5, 2013

As my server will never give me "a list of deleted records since last request" I need another solution.

For me "resetting" the store sounds like the easiest thing to do. However I didn't find such a thing in EmberData.
What would be the best way to do that ?

Member

tchak commented May 5, 2013

You should follow the discussion in #235. In some cases reseting the store is a good strategy.

Contributor

heartsentwined commented May 14, 2013

@tchak @ddewaele any status update on this? I ran into the push-deleted-record issue too, and I have already joined #235.

Like with many things Ember I'm never sure if it's a bug or a complete misunderstanding on my part :)

👍 That's the reason why I'm joining the discussion here too.

The way I see it, we cannot have ember auto-delete something client side, simply based on an absence from API responses. For small demo todo apps, it would make sense for the server to return, e.g. all items from a list, but venture even a bit into production, then I would say 99% of the use case will use pagination. The API will likely retrieve and return, e.g. the first 30 items, but not the entire list of 1000 items. Thus even for the same GET /api/items?page=1 query, any difference could be reasonably due to a newer item being created, thus pushing the 30th off - it is not deleted.

I am okay with making the server returning a list of deleted records, or clearing the data store altogether. However, I would lean towards the latter. The former is an expensive calculation server-side, essentially requiring it to diff all requests from all users; what's more, that's a diff that relies on a database operation (hence slow).

As for the former, is that currently implemented in ember-data? If yes, what is the expected format of such API responses?

four43 commented Jul 22, 2013

For anyone just coming into this discussion, this is apparent in the Getting Started To Do App:

http://emberjs.com/guides/getting-started/using-other-adapters/

If you flag an issue as completed, then hit the "Clear Completed" button, then click active, then all you will see the previously cleared issue re-appear. It does seem to save to the local storage correctly however because if you refresh the page it should stay deleted.

Owner

wagenet commented Aug 13, 2013

@cr125rider I'm not sure if this is actually the same bug, but it definitely is a bug.

four43 commented Aug 13, 2013

@wagenet - Ah, my apologies, they seemed similar to me. In the latest version of the libraries it seems to be fixed. I tested using the latest of each about 2 weeks ago. Might be good to keep that issue open and get a test case around it however. Thanks for taking a look at it.

Owner

wagenet commented Aug 13, 2013

@cr125rider Sounds good. We should at least make sure that it's working on the next guides update.

Owner

wycats commented Sep 5, 2013

It sounds like the request is to be able to say "I know that find() returns a list of all possible records, so if something isn't there, it's safe to remove locally".

This will work in non-paginated cases with a small data-set, but it doesn't seem like it'll work in a large number of practical cases. I'll entertain a PR that fleshed out the desired semantics here.

wycats closed this Sep 5, 2013

I was also facing the same issue (records deleted from the database were still showing at the front-end, as they were not removed from ember-data store).

The workaround I found for this is to send any arbitrary parameter along with the find() request which forces emberjs to load the records present only in the database.

e.g. assuming '/users' to be a route, the model hook I implemented for this route was as :
return this.store.find('user', {xyz: 'abc'}).then(onSuccess, onError);
[http://exampleWebServer/users?xyz=abc]

instead of :
return this.store.find('user').then(onSuccess, onError);
[http://exampleWebServer/users]

And now it's working as expected. I am not sure if this is the correct way to achieve it but it works just fine.

I'm running into the same problem. I simply want to refresh the store to reflect what's on the server.

The solution proposed by @sangramkapre works, but it should be noted that it's an ugly hack. Passing an arbitrary query object (or an empty object) will return the models reflected on the server, but the deleted models are still in the store. Here's an example to demonstrate this.

Let's say there are initially 5 users on the server:

this.store.find('user').then(
  function(d) { 
    console.log(d.get('length')); // 5
  }
);

One user gets deleted from the server.

Here's what happens when you pass in an empty query object:

this.store.find('user', {}).then(
  function(d) { 
    console.log(d.get('length')); // 4
  }
);

But note that this didn't refresh the store. The deleted user is still in there:

this.store.find('user').then(
  function(d) { 
    console.log(d.get('length')); // 5
  }
);

If you really want to refresh the store, you'll need to unload all of the data in the store before calling find():

this.store.unloadAll('user');
this.store.find('user').then(
  function(d) { 
    console.log(d.get('length')); // 4
  }
);

@johnnyoshika when i call unloadAll before reload the data from backend I see a 'flickering' on template that is displaying my model. How can I avoid the 'flickering'?

@cresg820, can you create a jsbin that demonstrates the problem?

cresg820 commented Feb 1, 2015

@johnnyoshika I create a simple demo at http://emberjs.jsbin.com/jebiqijimo/1/edit
When you click on refresh you see a fast blinking in template.

NotSqrt commented Mar 5, 2015

I have a similar situation : need to reflect in the local store that some remote objects are no longer there.

I am thinking of a mechanism : storing a "lastSync" date in each instance. After the refresh, if the "lastSync" is older that the moment where the client started the refresh, I can call "unloadRecord".

A better idea ?

Contributor

lolmaus commented Mar 5, 2015

My take based on @cresg820's jsbin: http://emberjs.jsbin.com/tinugi/3/edit?html,js,output

No flickering! :D

Be careful, it's gonna delete all records that do not exist on the backend, including those that the user might have created clientside but hasn't saved yet.

Yeah this is really depressing, I was banging my head against the wall for so so long ,and I figured it was something simple I was unable to do ,an action leads to an API call which deletes certain records in the server document and returns the updated document, but the model wont delete existing records...

I

Contributor

runspired commented Jul 22, 2015

@jordanblink it won't detect server side deletes, it knows about things you tell it you've deleted.

I am deleting the records on the server side and returning back the updated model as part of the response of the API call ,if I add records in the response , the records get added to the model but if I delete records and send the new document in the response, the model wont delete the record.

Contributor

runspired commented Jul 22, 2015

@jordanblink ping me in the ember community slack and I'll walk you through it.

Generate an Invite: https://ember-community-slackin.herokuapp.com/
Slack: embercommunity.slack.com

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment