Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LDS Performance Improvements #32

Closed
richardjrossiii opened this issue Aug 17, 2015 · 101 comments
Closed

LDS Performance Improvements #32

richardjrossiii opened this issue Aug 17, 2015 · 101 comments

Comments

@richardjrossiii
Copy link
Contributor

@richardjrossiii richardjrossiii commented Aug 17, 2015

This is a master issue for all things related to local datastore performance improvements.

  • Make LDS stop checkpointing and making copies of objects for mutable containers (#102)
  • Make LDS query matching more efficient by not loading every object into memory for a specified pin.
  • Make LDS use PFObjectState and PFUserState instead of PFObject and PFUser, to reduce the amount of locking required for every LDS operation.
  • Improve LDS PFRelation tracking.
  • Update LDS database schema to allow for query matching in the database layer itself.

For anyone reaching this issue as a result of currently bad LDS performance here's our current recommendations:

  • Keep your object schemas as small as possible - the faster it is to parse them from JSON, the better.
  • Relations are currently inefficient. If you can avoid them, do so.
  • Minimize usage of any values of the following classes:
    • NSArray
    • NSDictionary
    • PFACL
    • PFGeoPoint
  • Whenever possible, use smaller pins. Because LDS loads all objects in a single pin before evaluating a query, the smaller the pin is, the better.
@rawrmaan
Copy link

@rawrmaan rawrmaan commented Aug 19, 2015

+1, LDS performance needs lots of work. I'm glad you're finally publicly acknowledging this. See my angry bug report from April 2015: https://developers.facebook.com/bugs/1575754175998187

@FreudGit
Copy link

@FreudGit FreudGit commented Aug 25, 2015

+1: 2.3 sec to get one row on a table of 800 objects... Ouch.

@RickDT
Copy link

@RickDT RickDT commented Sep 1, 2015

👍

@FreudGit
Copy link

@FreudGit FreudGit commented Sep 3, 2015

Hi

Sorry to came back. Do you think one of these point can be associate to a milestone?

Maybe not clean, but a PFQuery with a whereKey pointing to a String can be filtered direct from the DB (json), without unwrap.

Just with this, Query time will be 99% smaller.

Just my 2 cents

@nlutsenko
Copy link
Contributor

@nlutsenko nlutsenko commented Sep 3, 2015

Hey @FreudGit, absolutely!
I've just attached a PR to this that implements the first bullet point in that list.
From our testing - it should improve performance significantly.

But you are right - we should attach more to milestones. Making it happen...

@FreudGit
Copy link

@FreudGit FreudGit commented Sep 3, 2015

WOW, thanks for the fast reply.

PR? This was a bit to technical for me. But i'm happy if we can target and see better performance in the next release\weeks. 👍

@nlutsenko
Copy link
Contributor

@nlutsenko nlutsenko commented Sep 3, 2015

PR is a shorthand for Pull Request

Since removing mutable containers functionality can break few flows for people, we are sitting on it for a little bit longer than next release. I think we are going to ship it with 1.8.4

@FreudGit
Copy link

@FreudGit FreudGit commented Sep 4, 2015

Thanks again @nlutsenko.
Wish 1.8.4 will come soon or that you will be ok to move to 1.8.3 milestone. Let me know if\when there is a way to try.

(completely off-topic)= how do you manage the multiple version(android, etc). Do you sync revision with other team(Android, .net, etc? Just curious!

@nlutsenko
Copy link
Contributor

@nlutsenko nlutsenko commented Sep 4, 2015

@FreudGit, 1.8.4 is coming very very soon after 1.8.4. A small tip is to look at the release schedule of 3 previous releases, it might get you an idea of how we try to release at the current moment.

So far we don't sync versions between Android/.NET/iOS(OSX), since we have separate dependencies.
If we are releasing a new feature that is cross-platform, say new Sessions APIs that we released at F8 - we do the release at the same time. Most of the time though - we have separate release schedules.

@FreudGit
Copy link

@FreudGit FreudGit commented Sep 4, 2015

Thanks(still curious). Are the dev of each language share code\need to have the more similar structure as possible?

@nlutsenko
Copy link
Contributor

@nlutsenko nlutsenko commented Sep 4, 2015

Not sure about the last question...
We don't share any code currently, but the APIs/class structure and architecture are unified and share a lot of the same principles and patterns, though adapted to best platform practices.

@FreudGit
Copy link

@FreudGit FreudGit commented Sep 4, 2015

Sorry for my bad english. And have a nice day.

Just curious if, for example, Android LDS have the same performance problem because very similair coding approach.

Dont want to bug anymore, no need to reply and no more question!

@nlutsenko
Copy link
Contributor

@nlutsenko nlutsenko commented Sep 4, 2015

Got you. I think the answer is yes - we share the same architecture and approaches for LDS, so say removing mutable containers on Android will also improve performance quite a lot!
No problem, please bug us, and I am always happy to help! 😁

@FreudGit
Copy link

@FreudGit FreudGit commented Sep 11, 2015

@nlutsenko Are 1.8.4 still a target for this ? :)

@nlutsenko
Copy link
Contributor

@nlutsenko nlutsenko commented Sep 11, 2015

I wish my answer was yes.
We are aiming for 1.8.4 for early/mid next week, so looks like 1.8.5 is the target for this.

@Mohd-Aamir-M
Copy link

@Mohd-Aamir-M Mohd-Aamir-M commented Oct 7, 2015

1.8.5 does contain good performance on LDS pinning and fetching?

And, LDS performance issue persists on other parse SDKs like Android, OSX etc.

@oskarl
Copy link

@oskarl oskarl commented Oct 7, 2015

Any update? Does anyone have tips for workarounds?

@Mohd-Aamir-M
Copy link

@Mohd-Aamir-M Mohd-Aamir-M commented Oct 9, 2015

I am glad to know that iOS sdk 1.9.0 is finally released so will it solve LDS issues?

@nlutsenko
Copy link
Contributor

@nlutsenko nlutsenko commented Oct 9, 2015

There are improvements for LDS, yes.
Solve - possibly, but #102 is marked for 1.10.0 - which will have a much bigger impact on the performance compared to 1.9.0.

@flovilmart
Copy link
Contributor

@flovilmart flovilmart commented Jul 16, 2017

That would be nice, thatMs always better to share the same architectures and patterns on both platforms. What do you think @dplewis?

@dplewis
Copy link
Member

@dplewis dplewis commented Jul 16, 2017

@flovilmart My app is on iOS and I use core data for a small set of data. I've been meaning to look into realm for a while now. I'm interested in the adapters

@eduardbosch
Copy link
Contributor

@eduardbosch eduardbosch commented Jul 19, 2017

@flovilmart what would be the next steps? Will start with iOS only right now?

I'll be happy to help in android 😃

@flovilmart
Copy link
Contributor

@flovilmart flovilmart commented Jul 19, 2017

Yeah, perhaps with and architectural proof of concept. Optimizing the storage for queries is where I believe we should go ie: creating additional tables with optimal indexes and then querying the table that holds the data by objectId. What do you think?

@eduardbosch
Copy link
Contributor

@eduardbosch eduardbosch commented Jul 20, 2017

This will speed up querying in LDS a lot. Could be a good start.


Anyway, I'm planning to move all my LDS to realm, so I can make queries directly to any data properties. Also, this will speed up a lot saving the data to LDS.

I was thinking about 3 options:

  1. Reuse classes that extend from ParseObject as RealmObject, so data could be serialized directly to realm and users don't have to convert their classes to realm DB. But I don't think that this is possible.
  2. Do it the opposite way, reuse the classes that extend from RealmObject so ParseQuery can serialize to this kind of objects.
  3. Create classes that extend directly from RealmObject, then query ParseObjects and do the conversion. When the conversion is done, save any extra fields existing in the ParseObjectnot defined in the RealmObject, as 3 arrays. An array of string keys, an array of string values and a third array that defines the types to allow deserializing. This arrays will allow saving any new fields.

Then, I was planning to use realm DB directly to make my queries.

To avoid dealing with realm DB, another improvement could be to translate the Parse queries to realm queries.

Those are just some thoughts about this topic.

What do you think?

Is this a good LDS option? Could be a viable solution?

I think the best solution is to have an automatic SQLite / CoreData like you are planning, so users will need only the Parse SDK. But to improve the LDS a lot more, I think that we can write an LDS abstraction that can be saved to some different kinds of DBs like realm or other.

@playvici
Copy link

@playvici playvici commented Jul 31, 2017

Are there any good ideas / success stories from others going from LDS to Realm? Core Data? Also didn't know LDS had been deprecated.

@eduardbosch
Copy link
Contributor

@eduardbosch eduardbosch commented Sep 5, 2017

@flovilmart @dplewis Any news on start a LDS alternative?

Thanks

@flovilmart
Copy link
Contributor

@flovilmart flovilmart commented Sep 5, 2017

@eduardbosch
Copy link
Contributor

@eduardbosch eduardbosch commented Jun 1, 2018

Hi to all again. I'm starting using the iOS lib and I'll have to implement an LDS.

Has anyone used any alternative to Parse LDS to combine the Parse SDK with Realm or another local database? I think I'll go with Realm as it allows working with lots of entries very efficiently.

I think that this is the biggest issue has Parse right now. With a good LDS, it'll be great.

@stale

This comment has been hidden.

@stale stale bot added the Stale label Sep 19, 2018
@stale stale bot closed this Sep 26, 2018
@TomWFox
Copy link
Member

@TomWFox TomWFox commented Mar 28, 2019

@eduardbosch what solution did you come to? I wonder whether offering a bridge between Parse and say Realm could be useful. Although we haven't had many recent reports I think that LDS is still not the best especially under high loads.

@eduardbosch
Copy link
Contributor

@eduardbosch eduardbosch commented Apr 1, 2019

@Tom-Fox1 I haven't implemented offline data on iOS and I don't know if I'll add it to my app in the near future. I think that LDS is not a good solution for high data DBs with large pins. It'll be really good to have a bridge between Parse and SQLite or Realm. That will make offline data access really shine.

Have you thought about any possible workaround?

@TomWFox
Copy link
Member

@TomWFox TomWFox commented Apr 1, 2019

@eduardbosch I’ve found that LDS has worked well for me, probably because I only need to pins a very small number of objects. However, I appreciate that it has problems and I see it as one of the most important issues to address with this SDK so I’m just looking into it.

@stale stale bot added the Stale label May 16, 2019
@TomWFox TomWFox added the Pinned label May 16, 2019
@stale stale bot removed the Stale label May 16, 2019
@parse-community parse-community deleted a comment from stale bot May 16, 2019
@TomWFox TomWFox removed the Pinned label Jul 3, 2020
@stale
Copy link

@stale stale bot commented Aug 18, 2020

This issue has been automatically marked as stale because it has not had recent activity. If you believe it should stay open, please let us know! As always, we encourage contributions, check out the Contributing Guide

@stale stale bot added the Stale label Aug 18, 2020
@stale stale bot closed this Aug 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet