New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LDS Performance Improvements #32

Open
richardjrossiii opened this Issue Aug 17, 2015 · 96 comments

Comments

Projects
None yet
@richardjrossiii
Contributor

richardjrossiii commented Aug 17, 2015

This is a master issue for all things related to local datastore performance improvements.

  • Make LDS stop checkpointing and making copies of objects for mutable containers (#102)
  • Make LDS query matching more efficient by not loading every object into memory for a specified pin.
  • Make LDS use PFObjectState and PFUserState instead of PFObject and PFUser, to reduce the amount of locking required for every LDS operation.
  • Improve LDS PFRelation tracking.
  • Update LDS database schema to allow for query matching in the database layer itself.

For anyone reaching this issue as a result of currently bad LDS performance here's our current recommendations:

  • Keep your object schemas as small as possible - the faster it is to parse them from JSON, the better.
  • Relations are currently inefficient. If you can avoid them, do so.
  • Minimize usage of any values of the following classes:
    • NSArray
    • NSDictionary
    • PFACL
    • PFGeoPoint
  • Whenever possible, use smaller pins. Because LDS loads all objects in a single pin before evaluating a query, the smaller the pin is, the better.
@rawrmaan

This comment has been minimized.

Show comment
Hide comment
@rawrmaan

rawrmaan Aug 19, 2015

+1, LDS performance needs lots of work. I'm glad you're finally publicly acknowledging this. See my angry bug report from April 2015: https://developers.facebook.com/bugs/1575754175998187

rawrmaan commented Aug 19, 2015

+1, LDS performance needs lots of work. I'm glad you're finally publicly acknowledging this. See my angry bug report from April 2015: https://developers.facebook.com/bugs/1575754175998187

@FreudGit

This comment has been minimized.

Show comment
Hide comment
@FreudGit

FreudGit Aug 25, 2015

+1: 2.3 sec to get one row on a table of 800 objects... Ouch.

FreudGit commented Aug 25, 2015

+1: 2.3 sec to get one row on a table of 800 objects... Ouch.

@RickDT

This comment has been minimized.

Show comment
Hide comment
@RickDT

RickDT commented Sep 1, 2015

👍

@FreudGit

This comment has been minimized.

Show comment
Hide comment
@FreudGit

FreudGit Sep 3, 2015

Hi

Sorry to came back. Do you think one of these point can be associate to a milestone?

Maybe not clean, but a PFQuery with a whereKey pointing to a String can be filtered direct from the DB (json), without unwrap.

Just with this, Query time will be 99% smaller.

Just my 2 cents

FreudGit commented Sep 3, 2015

Hi

Sorry to came back. Do you think one of these point can be associate to a milestone?

Maybe not clean, but a PFQuery with a whereKey pointing to a String can be filtered direct from the DB (json), without unwrap.

Just with this, Query time will be 99% smaller.

Just my 2 cents

@nlutsenko

This comment has been minimized.

Show comment
Hide comment
@nlutsenko

nlutsenko Sep 3, 2015

Contributor

Hey @FreudGit, absolutely!
I've just attached a PR to this that implements the first bullet point in that list.
From our testing - it should improve performance significantly.

But you are right - we should attach more to milestones. Making it happen...

Contributor

nlutsenko commented Sep 3, 2015

Hey @FreudGit, absolutely!
I've just attached a PR to this that implements the first bullet point in that list.
From our testing - it should improve performance significantly.

But you are right - we should attach more to milestones. Making it happen...

@FreudGit

This comment has been minimized.

Show comment
Hide comment
@FreudGit

FreudGit Sep 3, 2015

WOW, thanks for the fast reply.

PR? This was a bit to technical for me. But i'm happy if we can target and see better performance in the next release\weeks. 👍

FreudGit commented Sep 3, 2015

WOW, thanks for the fast reply.

PR? This was a bit to technical for me. But i'm happy if we can target and see better performance in the next release\weeks. 👍

@nlutsenko

This comment has been minimized.

Show comment
Hide comment
@nlutsenko

nlutsenko Sep 3, 2015

Contributor

PR is a shorthand for Pull Request

Since removing mutable containers functionality can break few flows for people, we are sitting on it for a little bit longer than next release. I think we are going to ship it with 1.8.4

Contributor

nlutsenko commented Sep 3, 2015

PR is a shorthand for Pull Request

Since removing mutable containers functionality can break few flows for people, we are sitting on it for a little bit longer than next release. I think we are going to ship it with 1.8.4

@FreudGit

This comment has been minimized.

Show comment
Hide comment
@FreudGit

FreudGit Sep 4, 2015

Thanks again @nlutsenko.
Wish 1.8.4 will come soon or that you will be ok to move to 1.8.3 milestone. Let me know if\when there is a way to try.

(completely off-topic)= how do you manage the multiple version(android, etc). Do you sync revision with other team(Android, .net, etc? Just curious!

FreudGit commented Sep 4, 2015

Thanks again @nlutsenko.
Wish 1.8.4 will come soon or that you will be ok to move to 1.8.3 milestone. Let me know if\when there is a way to try.

(completely off-topic)= how do you manage the multiple version(android, etc). Do you sync revision with other team(Android, .net, etc? Just curious!

@nlutsenko

This comment has been minimized.

Show comment
Hide comment
@nlutsenko

nlutsenko Sep 4, 2015

Contributor

@FreudGit, 1.8.4 is coming very very soon after 1.8.4. A small tip is to look at the release schedule of 3 previous releases, it might get you an idea of how we try to release at the current moment.

So far we don't sync versions between Android/.NET/iOS(OSX), since we have separate dependencies.
If we are releasing a new feature that is cross-platform, say new Sessions APIs that we released at F8 - we do the release at the same time. Most of the time though - we have separate release schedules.

Contributor

nlutsenko commented Sep 4, 2015

@FreudGit, 1.8.4 is coming very very soon after 1.8.4. A small tip is to look at the release schedule of 3 previous releases, it might get you an idea of how we try to release at the current moment.

So far we don't sync versions between Android/.NET/iOS(OSX), since we have separate dependencies.
If we are releasing a new feature that is cross-platform, say new Sessions APIs that we released at F8 - we do the release at the same time. Most of the time though - we have separate release schedules.

@FreudGit

This comment has been minimized.

Show comment
Hide comment
@FreudGit

FreudGit Sep 4, 2015

Thanks(still curious). Are the dev of each language share code\need to have the more similar structure as possible?

FreudGit commented Sep 4, 2015

Thanks(still curious). Are the dev of each language share code\need to have the more similar structure as possible?

@nlutsenko

This comment has been minimized.

Show comment
Hide comment
@nlutsenko

nlutsenko Sep 4, 2015

Contributor

Not sure about the last question...
We don't share any code currently, but the APIs/class structure and architecture are unified and share a lot of the same principles and patterns, though adapted to best platform practices.

Contributor

nlutsenko commented Sep 4, 2015

Not sure about the last question...
We don't share any code currently, but the APIs/class structure and architecture are unified and share a lot of the same principles and patterns, though adapted to best platform practices.

@FreudGit

This comment has been minimized.

Show comment
Hide comment
@FreudGit

FreudGit Sep 4, 2015

Sorry for my bad english. And have a nice day.

Just curious if, for example, Android LDS have the same performance problem because very similair coding approach.

Dont want to bug anymore, no need to reply and no more question!

FreudGit commented Sep 4, 2015

Sorry for my bad english. And have a nice day.

Just curious if, for example, Android LDS have the same performance problem because very similair coding approach.

Dont want to bug anymore, no need to reply and no more question!

@nlutsenko

This comment has been minimized.

Show comment
Hide comment
@nlutsenko

nlutsenko Sep 4, 2015

Contributor

Got you. I think the answer is yes - we share the same architecture and approaches for LDS, so say removing mutable containers on Android will also improve performance quite a lot!
No problem, please bug us, and I am always happy to help! 😁

Contributor

nlutsenko commented Sep 4, 2015

Got you. I think the answer is yes - we share the same architecture and approaches for LDS, so say removing mutable containers on Android will also improve performance quite a lot!
No problem, please bug us, and I am always happy to help! 😁

@FreudGit

This comment has been minimized.

Show comment
Hide comment
@FreudGit

FreudGit Sep 11, 2015

@nlutsenko Are 1.8.4 still a target for this ? :)

FreudGit commented Sep 11, 2015

@nlutsenko Are 1.8.4 still a target for this ? :)

@nlutsenko

This comment has been minimized.

Show comment
Hide comment
@nlutsenko

nlutsenko Sep 11, 2015

Contributor

I wish my answer was yes.
We are aiming for 1.8.4 for early/mid next week, so looks like 1.8.5 is the target for this.

Contributor

nlutsenko commented Sep 11, 2015

I wish my answer was yes.
We are aiming for 1.8.4 for early/mid next week, so looks like 1.8.5 is the target for this.

@PSIPhone

This comment has been minimized.

Show comment
Hide comment
@PSIPhone

PSIPhone Oct 7, 2015

1.8.5 does contain good performance on LDS pinning and fetching?

And, LDS performance issue persists on other parse SDKs like Android, OSX etc.

PSIPhone commented Oct 7, 2015

1.8.5 does contain good performance on LDS pinning and fetching?

And, LDS performance issue persists on other parse SDKs like Android, OSX etc.

@oskarl

This comment has been minimized.

Show comment
Hide comment
@oskarl

oskarl Oct 7, 2015

Any update? Does anyone have tips for workarounds?

oskarl commented Oct 7, 2015

Any update? Does anyone have tips for workarounds?

@PSIPhone

This comment has been minimized.

Show comment
Hide comment
@PSIPhone

PSIPhone Oct 9, 2015

I am glad to know that iOS sdk 1.9.0 is finally released so will it solve LDS issues?

PSIPhone commented Oct 9, 2015

I am glad to know that iOS sdk 1.9.0 is finally released so will it solve LDS issues?

@nlutsenko

This comment has been minimized.

Show comment
Hide comment
@nlutsenko

nlutsenko Oct 9, 2015

Contributor

There are improvements for LDS, yes.
Solve - possibly, but #102 is marked for 1.10.0 - which will have a much bigger impact on the performance compared to 1.9.0.

Contributor

nlutsenko commented Oct 9, 2015

There are improvements for LDS, yes.
Solve - possibly, but #102 is marked for 1.10.0 - which will have a much bigger impact on the performance compared to 1.9.0.

@PSIPhone

This comment has been minimized.

Show comment
Hide comment
@PSIPhone

PSIPhone Oct 9, 2015

so 1.9.0 will solve slow pinning issue in LDS? And 1.10.0 will solve mutable container isssue right?

I think i should update to 1.9.0

PSIPhone commented Oct 9, 2015

so 1.9.0 will solve slow pinning issue in LDS? And 1.10.0 will solve mutable container isssue right?

I think i should update to 1.9.0

@nlutsenko

This comment has been minimized.

Show comment
Hide comment
@nlutsenko

nlutsenko Oct 9, 2015

Contributor

1.10.0 will remove mutable containers, which degrade LDS performance by a significant portion of time.

Contributor

nlutsenko commented Oct 9, 2015

1.10.0 will remove mutable containers, which degrade LDS performance by a significant portion of time.

@PSIPhone

This comment has been minimized.

Show comment
Hide comment
@PSIPhone

PSIPhone Oct 9, 2015

So for slow pinning 1.9.0 is perfect solution for now?

And what are the mutable containers i don't find anything about it in docs.

PSIPhone commented Oct 9, 2015

So for slow pinning 1.9.0 is perfect solution for now?

And what are the mutable containers i don't find anything about it in docs.

@nlutsenko

This comment has been minimized.

Show comment
Hide comment
@nlutsenko

nlutsenko Oct 9, 2015

Contributor

#97 - has the information on what's changing.
The latest framework version always has the latest and most up-to-date implementations of everything.

Contributor

nlutsenko commented Oct 9, 2015

#97 - has the information on what's changing.
The latest framework version always has the latest and most up-to-date implementations of everything.

@stoutj

This comment has been minimized.

Show comment
Hide comment
@stoutj

stoutj Oct 9, 2015

Can we get an update on the Android version? When can we expect the LOS performance issue to be solved on Android?

stoutj commented Oct 9, 2015

Can we get an update on the Android version? When can we expect the LOS performance issue to be solved on Android?

@nlutsenko

This comment has been minimized.

Show comment
Hide comment
@nlutsenko

nlutsenko Oct 9, 2015

Contributor

cc @grantland for Android version.

Contributor

nlutsenko commented Oct 9, 2015

cc @grantland for Android version.

@FreudGit

This comment has been minimized.

Show comment
Hide comment
@FreudGit

FreudGit Oct 29, 2015

@nlutsenko What are the next checkbox to be solved? :)

FreudGit commented Oct 29, 2015

@nlutsenko What are the next checkbox to be solved? :)

@nlutsenko

This comment has been minimized.

Show comment
Hide comment
@nlutsenko

nlutsenko Nov 30, 2015

Contributor

@kevflynn That's a good one, indeed. Not sure on how feasible this is with our schema though.
Non-sqlite built in indices is a good option here, but would likely require a lot of work.
I found easier to grasp things, like the one surfaced in #605

Contributor

nlutsenko commented Nov 30, 2015

@kevflynn That's a good one, indeed. Not sure on how feasible this is with our schema though.
Non-sqlite built in indices is a good option here, but would likely require a lot of work.
I found easier to grasp things, like the one surfaced in #605

@FreudGit

This comment has been minimized.

Show comment
Hide comment
@FreudGit

FreudGit Dec 2, 2015

What's next? :)

FreudGit commented Dec 2, 2015

What's next? :)

@nlutsenko

This comment has been minimized.

Show comment
Hide comment
@nlutsenko

nlutsenko Dec 2, 2015

Contributor

@FreudGit Actually, more of the same (as #605), batching stuff together proves to be very helpful in regards to performance, and there is way more we can do in this land. 😉

Contributor

nlutsenko commented Dec 2, 2015

@FreudGit Actually, more of the same (as #605), batching stuff together proves to be very helpful in regards to performance, and there is way more we can do in this land. 😉

@grantland grantland referenced this issue Dec 3, 2015

Open

LDS Performance Improvements #279

1 of 5 tasks complete
@rossrossp

This comment has been minimized.

Show comment
Hide comment
@rossrossp

rossrossp Jan 10, 2016

I'm still experiencing slow response times on simple queries with the latest SDK. What's the latest?

rossrossp commented Jan 10, 2016

I'm still experiencing slow response times on simple queries with the latest SDK. What's the latest?

@FreudGit

This comment has been minimized.

Show comment
Hide comment
@FreudGit

FreudGit commented Jan 13, 2016

+1

@sobri909

This comment has been minimized.

Show comment
Hide comment
@sobri909

sobri909 Jan 23, 2016

  • Make LDS query matching more efficient by not loading every object into memory for a specified pin.
  • Update LDS database schema to allow for query matching in the database layer itself.

These two are on the knife edge of deal breakers for me. I do wish I'd been aware of these limitations earlier on.

Is any progress being made on these optimisations? I don't want to sound combative, but calling them optimisations is euphemistic. A database layer that requires loading every object into memory to perform a query, and has no indexing, thus needs to evaluate every object directly ... that's quite something.

Even if the other release blocker issues I'm having are resolved (#768, #772), those two points listed above are leaving me with the realisation that I may have made a terrible mistake.

Particularly #768 is exposing the fatal limitations of the above two points. Failure to unpin is causing a regularly queried pin to grow to thousands of objects over the course of a day, bringing the app to a halt. Simple queries of a table of ~4000 rows should not bring a database to its knees.

Basically I'm looking for some show of confidence that these limitations are actually being worked on, and there is some timeline to seeing them resolved. Otherwise I have to make the painful decision of jumping ship.

I very very much don't want to have to do that after investing several months in a project based on Parse. Especially considering the Parse SDK public APIs (both client and server side) are rather good. They are well designed. They expose smart and powerful functionality through plainly readable and elegant APIs. But the inner limitations and failings I've run up against, both client and server side (though this isn't the place to discuss the latter) have pushed things into the "bad business decision" territory.

Are there any assurances you can give that the above two fixes are going to happen? And in a reasonable timeframe? Are they being actively worked on?

I don't want to jump ship. That would be a timeline disaster for me, and I'm not feeling drawn to any of the competitor's offerings. But I'm reaching the "unpleasant decisions" point.

sobri909 commented Jan 23, 2016

  • Make LDS query matching more efficient by not loading every object into memory for a specified pin.
  • Update LDS database schema to allow for query matching in the database layer itself.

These two are on the knife edge of deal breakers for me. I do wish I'd been aware of these limitations earlier on.

Is any progress being made on these optimisations? I don't want to sound combative, but calling them optimisations is euphemistic. A database layer that requires loading every object into memory to perform a query, and has no indexing, thus needs to evaluate every object directly ... that's quite something.

Even if the other release blocker issues I'm having are resolved (#768, #772), those two points listed above are leaving me with the realisation that I may have made a terrible mistake.

Particularly #768 is exposing the fatal limitations of the above two points. Failure to unpin is causing a regularly queried pin to grow to thousands of objects over the course of a day, bringing the app to a halt. Simple queries of a table of ~4000 rows should not bring a database to its knees.

Basically I'm looking for some show of confidence that these limitations are actually being worked on, and there is some timeline to seeing them resolved. Otherwise I have to make the painful decision of jumping ship.

I very very much don't want to have to do that after investing several months in a project based on Parse. Especially considering the Parse SDK public APIs (both client and server side) are rather good. They are well designed. They expose smart and powerful functionality through plainly readable and elegant APIs. But the inner limitations and failings I've run up against, both client and server side (though this isn't the place to discuss the latter) have pushed things into the "bad business decision" territory.

Are there any assurances you can give that the above two fixes are going to happen? And in a reasonable timeframe? Are they being actively worked on?

I don't want to jump ship. That would be a timeline disaster for me, and I'm not feeling drawn to any of the competitor's offerings. But I'm reaching the "unpleasant decisions" point.

@sobri909

This comment has been minimized.

Show comment
Hide comment
@sobri909

sobri909 Jan 23, 2016

Aside: I will be working on reproducing #768 in a sample project today, to help that one along, because it is a release blocker and I'm due to beta in a week. Apologies for not having one ready yet, but today is hopefully the day. But even once that one is resolved, the above fundamental performance issues will still remain, leaving the same unpleasant decision to be made.

sobri909 commented Jan 23, 2016

Aside: I will be working on reproducing #768 in a sample project today, to help that one along, because it is a release blocker and I'm due to beta in a week. Apologies for not having one ready yet, but today is hopefully the day. But even once that one is resolved, the above fundamental performance issues will still remain, leaving the same unpleasant decision to be made.

@oskarl

This comment has been minimized.

Show comment
Hide comment
@oskarl

oskarl Jan 24, 2016

@sobri909 I switched to using CoreData (through the AlecrimCoreData wrapper) for the local side of my app and it solved a lot of issues for me without taking a lot of time. Basically I just wrote some methods for manually "pinning" parse objects.

oskarl commented Jan 24, 2016

@sobri909 I switched to using CoreData (through the AlecrimCoreData wrapper) for the local side of my app and it solved a lot of issues for me without taking a lot of time. Basically I just wrote some methods for manually "pinning" parse objects.

@rawrmaan

This comment has been minimized.

Show comment
Hide comment
@rawrmaan

rawrmaan Jan 26, 2016

@sobri909 I would recommend jumping ship. I gave the team a detailed analysis and harsh reply in May of last year and they haven't done anything about it. They simply don't care about this issue. I'm sorry about your timeline :(

My report: https://developers.facebook.com/bugs/1575754175998187/

rawrmaan commented Jan 26, 2016

@sobri909 I would recommend jumping ship. I gave the team a detailed analysis and harsh reply in May of last year and they haven't done anything about it. They simply don't care about this issue. I'm sorry about your timeline :(

My report: https://developers.facebook.com/bugs/1575754175998187/

@sobri909

This comment has been minimized.

Show comment
Hide comment
@sobri909

sobri909 Jan 27, 2016

@rawrmaan Yes, it's very disappointing.

I'm going to work around it as best I can for beta, but post-beta I'll begin the process of moving away from Parse entirely.

If it were only either issues on the client side or the server side, then sticking with Parse could possibly be justified, while working around specific limitations. But having run into fundamental shortcomings on both client and server, and a seeming lack of interest in remedying them, it's time to make unpleasant decisions.

To the Parse team, the least you could do would be to add a warning to the documentation that the Local Data Store should not be used for more than a few hundred objects. You would save developers from wasting time on doomed implementations.

sobri909 commented Jan 27, 2016

@rawrmaan Yes, it's very disappointing.

I'm going to work around it as best I can for beta, but post-beta I'll begin the process of moving away from Parse entirely.

If it were only either issues on the client side or the server side, then sticking with Parse could possibly be justified, while working around specific limitations. But having run into fundamental shortcomings on both client and server, and a seeming lack of interest in remedying them, it's time to make unpleasant decisions.

To the Parse team, the least you could do would be to add a warning to the documentation that the Local Data Store should not be used for more than a few hundred objects. You would save developers from wasting time on doomed implementations.

@FreudGit

This comment has been minimized.

Show comment
Hide comment
@FreudGit

FreudGit Jan 27, 2016

@nlutsenko are always very open and proactive. Can we please have some feedback about what are the next steps?

2 suggestions:

1- "experimental" query.fromLocalDatastoreAsDirectAccessToDB() (you may just accept Int and String query to start, so more safe)

2- Update LDS database schema to allow for query matching in the database layer itself.

FreudGit commented Jan 27, 2016

@nlutsenko are always very open and proactive. Can we please have some feedback about what are the next steps?

2 suggestions:

1- "experimental" query.fromLocalDatastoreAsDirectAccessToDB() (you may just accept Int and String query to start, so more safe)

2- Update LDS database schema to allow for query matching in the database layer itself.

@sobri909

This comment has been minimized.

Show comment
Hide comment
@sobri909

sobri909 Jan 28, 2016

Looking at the current SQLite database structure, I think one quick improvement would be to change pins to be actual SQL relations instead of ... well, what they are now. The current shape of pins in the SQL layer looks like it'd be a considerable bottleneck, and quite possibly a cause of race condition bugs.

That wouldn't solve the two serious points mentioned above, but it does look like low hanging fruit that could have reasonable payoff in the short term, and could be automatically migrated easily enough for existing installs.

sobri909 commented Jan 28, 2016

Looking at the current SQLite database structure, I think one quick improvement would be to change pins to be actual SQL relations instead of ... well, what they are now. The current shape of pins in the SQL layer looks like it'd be a considerable bottleneck, and quite possibly a cause of race condition bugs.

That wouldn't solve the two serious points mentioned above, but it does look like low hanging fruit that could have reasonable payoff in the short term, and could be automatically migrated easily enough for existing installs.

@sobri909

This comment has been minimized.

Show comment
Hide comment
@sobri909

sobri909 Jan 28, 2016

It also seems to me that, while in some cases they are useful, named pins are in general an added complexity with more negatives than positives. In an ideal world of a highly performant LDS, I think I would deprecate named pins.

The structure of a well designed database schema coupled with an efficient querying system should be adequate. Any required functionality similar to named pins could / should be added to a schema directly, on a per app basis. And the reduced complexity would be beneficial both to the SDK team(s) and to the users of.

But that's getting way ahead of things. At this stage I think to make any forward progress at all the changes need to be non breaking and relatively atomic. From the discussion above it sounds like the SDK team are essentially blocked on anything that requires larger migrations or backwards compat breaks.

sobri909 commented Jan 28, 2016

It also seems to me that, while in some cases they are useful, named pins are in general an added complexity with more negatives than positives. In an ideal world of a highly performant LDS, I think I would deprecate named pins.

The structure of a well designed database schema coupled with an efficient querying system should be adequate. Any required functionality similar to named pins could / should be added to a schema directly, on a per app basis. And the reduced complexity would be beneficial both to the SDK team(s) and to the users of.

But that's getting way ahead of things. At this stage I think to make any forward progress at all the changes need to be non breaking and relatively atomic. From the discussion above it sounds like the SDK team are essentially blocked on anything that requires larger migrations or backwards compat breaks.

@sobri909

This comment has been minimized.

Show comment
Hide comment
@sobri909

sobri909 commented Jan 29, 2016

http://blog.parse.com/announcements/moving-on/

Well that explains a lot. Maybe.

@FreudGit

This comment has been minimized.

Show comment
Hide comment
@FreudGit

FreudGit Jan 29, 2016

Yes, very sad.

FreudGit commented Jan 29, 2016

Yes, very sad.

@richardjrossiii

This comment has been minimized.

Show comment
Hide comment
@richardjrossiii

richardjrossiii Jan 29, 2016

Contributor

@sobri909 @FreudGit Parse is not going away, technically. We have just recently open sourced a compatible back-end for all of our client SDKs (https://github.com/parseplatform/parse-server), and all of these open-source projects will still be maintained well into the future.

There's been some awesome community effort as well to get things like dashboard, as well as one-click heroku hosting.

There's also several killer SDK features that still have yet to be rolled out, so stay tuned! Parse is dead, long live Parse!

Contributor

richardjrossiii commented Jan 29, 2016

@sobri909 @FreudGit Parse is not going away, technically. We have just recently open sourced a compatible back-end for all of our client SDKs (https://github.com/parseplatform/parse-server), and all of these open-source projects will still be maintained well into the future.

There's been some awesome community effort as well to get things like dashboard, as well as one-click heroku hosting.

There's also several killer SDK features that still have yet to be rolled out, so stay tuned! Parse is dead, long live Parse!

@sobri909

This comment has been minimized.

Show comment
Hide comment
@sobri909

sobri909 Jan 30, 2016

@richardjrosiii it's comforting to see how well the transition is being planned, especially the long timeline. Very admirable. But the above issues still remain and there's been no show of commitment from the current SDK team to resolve them.

As it stands, Parse on iOS is unusable for anyone who has local storage requirements beyond a few hundred objects.

The fastest way around that limitation is to use something else for local storage (eg Core Data) and bridge between. It's taken me a couple of days to put that together and I've achieved a speed boost from previously tens of seconds to get results to now getting results effectively instantly.

If I could see a quick path to achieving similar improvements in the Parse SDK itself I'd contribute them, but with the current state of the implementation I just don't think it's there. It's looking like potentially weeks of work. Effectively a ground up rewrite of the LDS.

It's a shame, but the smartest course of action still appears to be to jump ship.

sobri909 commented Jan 30, 2016

@richardjrosiii it's comforting to see how well the transition is being planned, especially the long timeline. Very admirable. But the above issues still remain and there's been no show of commitment from the current SDK team to resolve them.

As it stands, Parse on iOS is unusable for anyone who has local storage requirements beyond a few hundred objects.

The fastest way around that limitation is to use something else for local storage (eg Core Data) and bridge between. It's taken me a couple of days to put that together and I've achieved a speed boost from previously tens of seconds to get results to now getting results effectively instantly.

If I could see a quick path to achieving similar improvements in the Parse SDK itself I'd contribute them, but with the current state of the implementation I just don't think it's there. It's looking like potentially weeks of work. Effectively a ground up rewrite of the LDS.

It's a shame, but the smartest course of action still appears to be to jump ship.

@Silkjaer

This comment has been minimized.

Show comment
Hide comment
@Silkjaer

Silkjaer Jun 16, 2016

Maybe it would be beneficial to look at an integration with realm.io?

Silkjaer commented Jun 16, 2016

Maybe it would be beneficial to look at an integration with realm.io?

@jameslin101

This comment has been minimized.

Show comment
Hide comment
@jameslin101

jameslin101 Sep 16, 2016

@sobri909 I'm also getting 10s of seconds of load time from local datastore. Would you be able elaborate a little on what you did to do the bridging? Thanks in advance!

@nlutsenko is there still development on improving the LDS? seems like the thread is dead now

jameslin101 commented Sep 16, 2016

@sobri909 I'm also getting 10s of seconds of load time from local datastore. Would you be able elaborate a little on what you did to do the bridging? Thanks in advance!

@nlutsenko is there still development on improving the LDS? seems like the thread is dead now

@sobri909

This comment has been minimized.

Show comment
Hide comment
@sobri909

sobri909 Sep 16, 2016

@jameslin101 I moved to a combination of CloudKit and Core Data.

Parse's local storage implementation is effectively unusable. So taking Parse without local storage (thus requiring manual bridging to Core Data) and comparing it to CloudKit bridged with Core Data, the pros and cons weigh in CloudKit's favour.

Using Core Data for local storage, I gave each model class a ckRecord property that holds its corresponding CloudKit record.

When the Core Data context is saved I update a saveDate on each object. When a Core Data object is synced to CloudKit I update a ckSaveDate on the object. That makes it easy to find all Core Data objects that require syncing to CloudKit (ie their ckSaveDate is either nil or prior to saveDate).

For syncing changes from CloudKit to Core Data, that's a bit more complex. But generally the starting point (except for initial install/restore) is to use CloudKit subscriptions.

sobri909 commented Sep 16, 2016

@jameslin101 I moved to a combination of CloudKit and Core Data.

Parse's local storage implementation is effectively unusable. So taking Parse without local storage (thus requiring manual bridging to Core Data) and comparing it to CloudKit bridged with Core Data, the pros and cons weigh in CloudKit's favour.

Using Core Data for local storage, I gave each model class a ckRecord property that holds its corresponding CloudKit record.

When the Core Data context is saved I update a saveDate on each object. When a Core Data object is synced to CloudKit I update a ckSaveDate on the object. That makes it easy to find all Core Data objects that require syncing to CloudKit (ie their ckSaveDate is either nil or prior to saveDate).

For syncing changes from CloudKit to Core Data, that's a bit more complex. But generally the starting point (except for initial install/restore) is to use CloudKit subscriptions.

@jameslin101

This comment has been minimized.

Show comment
Hide comment
@jameslin101

jameslin101 commented Sep 16, 2016

@sobri909 Thank you!

@steve21124

This comment has been minimized.

Show comment
Hide comment
@steve21124

steve21124 Nov 14, 2016

Integrating the to realm.io will be a good idea. I have checkout the database entry

steve21124 commented Nov 14, 2016

Integrating the to realm.io will be a good idea. I have checkout the database entry

@eduardbosch

This comment has been minimized.

Show comment
Hide comment
@eduardbosch

eduardbosch Jul 12, 2017

I was thinking also about moving my LDS data to Realm db, which is blazing fast.

Will be really nice to have a useful LDS implementation that allows storing thousands of records.

eduardbosch commented Jul 12, 2017

I was thinking also about moving my LDS data to Realm db, which is blazing fast.

Will be really nice to have a useful LDS implementation that allows storing thousands of records.

@flovilmart

This comment has been minimized.

Show comment
Hide comment
@flovilmart

flovilmart Jul 16, 2017

Member

Would you guys be interesting in implementing a LDS adapter interface so one would be able to register it’s custom adapter and use CoreData, CK, or realm? I’ll gladly help getting you started :)

Member

flovilmart commented Jul 16, 2017

Would you guys be interesting in implementing a LDS adapter interface so one would be able to register it’s custom adapter and use CoreData, CK, or realm? I’ll gladly help getting you started :)

@dplewis

This comment has been minimized.

Show comment
Hide comment
@dplewis

dplewis Jul 16, 2017

Member

+1

Member

dplewis commented Jul 16, 2017

+1

@eduardbosch

This comment has been minimized.

Show comment
Hide comment
@eduardbosch

eduardbosch Jul 16, 2017

@flovilmart I'm really interested. My app is only for Android platform right now, so I'll like to implement the adapter, but for android 😊

I was planning to create a simple adapter for my app needs, but I'll prefer to implement it for the Parse SDK 😍

eduardbosch commented Jul 16, 2017

@flovilmart I'm really interested. My app is only for Android platform right now, so I'll like to implement the adapter, but for android 😊

I was planning to create a simple adapter for my app needs, but I'll prefer to implement it for the Parse SDK 😍

@flovilmart

This comment has been minimized.

Show comment
Hide comment
@flovilmart

flovilmart Jul 16, 2017

Member

That would be nice, thatMs always better to share the same architectures and patterns on both platforms. What do you think @dplewis?

Member

flovilmart commented Jul 16, 2017

That would be nice, thatMs always better to share the same architectures and patterns on both platforms. What do you think @dplewis?

@dplewis

This comment has been minimized.

Show comment
Hide comment
@dplewis

dplewis Jul 16, 2017

Member

@flovilmart My app is on iOS and I use core data for a small set of data. I've been meaning to look into realm for a while now. I'm interested in the adapters

Member

dplewis commented Jul 16, 2017

@flovilmart My app is on iOS and I use core data for a small set of data. I've been meaning to look into realm for a while now. I'm interested in the adapters

@eduardbosch

This comment has been minimized.

Show comment
Hide comment
@eduardbosch

eduardbosch Jul 19, 2017

@flovilmart what would be the next steps? Will start with iOS only right now?

I'll be happy to help in android 😃

eduardbosch commented Jul 19, 2017

@flovilmart what would be the next steps? Will start with iOS only right now?

I'll be happy to help in android 😃

@flovilmart

This comment has been minimized.

Show comment
Hide comment
@flovilmart

flovilmart Jul 19, 2017

Member

Yeah, perhaps with and architectural proof of concept. Optimizing the storage for queries is where I believe we should go ie: creating additional tables with optimal indexes and then querying the table that holds the data by objectId. What do you think?

Member

flovilmart commented Jul 19, 2017

Yeah, perhaps with and architectural proof of concept. Optimizing the storage for queries is where I believe we should go ie: creating additional tables with optimal indexes and then querying the table that holds the data by objectId. What do you think?

@eduardbosch

This comment has been minimized.

Show comment
Hide comment
@eduardbosch

eduardbosch Jul 20, 2017

This will speed up querying in LDS a lot. Could be a good start.


Anyway, I'm planning to move all my LDS to realm, so I can make queries directly to any data properties. Also, this will speed up a lot saving the data to LDS.

I was thinking about 3 options:

  1. Reuse classes that extend from ParseObject as RealmObject, so data could be serialized directly to realm and users don't have to convert their classes to realm DB. But I don't think that this is possible.
  2. Do it the opposite way, reuse the classes that extend from RealmObject so ParseQuery can serialize to this kind of objects.
  3. Create classes that extend directly from RealmObject, then query ParseObjects and do the conversion. When the conversion is done, save any extra fields existing in the ParseObjectnot defined in the RealmObject, as 3 arrays. An array of string keys, an array of string values and a third array that defines the types to allow deserializing. This arrays will allow saving any new fields.

Then, I was planning to use realm DB directly to make my queries.

To avoid dealing with realm DB, another improvement could be to translate the Parse queries to realm queries.

Those are just some thoughts about this topic.

What do you think?

Is this a good LDS option? Could be a viable solution?

I think the best solution is to have an automatic SQLite / CoreData like you are planning, so users will need only the Parse SDK. But to improve the LDS a lot more, I think that we can write an LDS abstraction that can be saved to some different kinds of DBs like realm or other.

eduardbosch commented Jul 20, 2017

This will speed up querying in LDS a lot. Could be a good start.


Anyway, I'm planning to move all my LDS to realm, so I can make queries directly to any data properties. Also, this will speed up a lot saving the data to LDS.

I was thinking about 3 options:

  1. Reuse classes that extend from ParseObject as RealmObject, so data could be serialized directly to realm and users don't have to convert their classes to realm DB. But I don't think that this is possible.
  2. Do it the opposite way, reuse the classes that extend from RealmObject so ParseQuery can serialize to this kind of objects.
  3. Create classes that extend directly from RealmObject, then query ParseObjects and do the conversion. When the conversion is done, save any extra fields existing in the ParseObjectnot defined in the RealmObject, as 3 arrays. An array of string keys, an array of string values and a third array that defines the types to allow deserializing. This arrays will allow saving any new fields.

Then, I was planning to use realm DB directly to make my queries.

To avoid dealing with realm DB, another improvement could be to translate the Parse queries to realm queries.

Those are just some thoughts about this topic.

What do you think?

Is this a good LDS option? Could be a viable solution?

I think the best solution is to have an automatic SQLite / CoreData like you are planning, so users will need only the Parse SDK. But to improve the LDS a lot more, I think that we can write an LDS abstraction that can be saved to some different kinds of DBs like realm or other.

@playvici

This comment has been minimized.

Show comment
Hide comment
@playvici

playvici Jul 31, 2017

Are there any good ideas / success stories from others going from LDS to Realm? Core Data? Also didn't know LDS had been deprecated.

playvici commented Jul 31, 2017

Are there any good ideas / success stories from others going from LDS to Realm? Core Data? Also didn't know LDS had been deprecated.

@eduardbosch

This comment has been minimized.

Show comment
Hide comment
@eduardbosch

eduardbosch Sep 5, 2017

@flovilmart @dplewis Any news on start a LDS alternative?

Thanks

eduardbosch commented Sep 5, 2017

@flovilmart @dplewis Any news on start a LDS alternative?

Thanks

@flovilmart

This comment has been minimized.

Show comment
Hide comment
@flovilmart

flovilmart Sep 5, 2017

Member
Member

flovilmart commented Sep 5, 2017

@eduardbosch

This comment has been minimized.

Show comment
Hide comment
@eduardbosch

eduardbosch Jun 1, 2018

Hi to all again. I'm starting using the iOS lib and I'll have to implement an LDS.

Has anyone used any alternative to Parse LDS to combine the Parse SDK with Realm or another local database? I think I'll go with Realm as it allows working with lots of entries very efficiently.

I think that this is the biggest issue has Parse right now. With a good LDS, it'll be great.

eduardbosch commented Jun 1, 2018

Hi to all again. I'm starting using the iOS lib and I'll have to implement an LDS.

Has anyone used any alternative to Parse LDS to combine the Parse SDK with Realm or another local database? I think I'll go with Realm as it allows working with lots of entries very efficiently.

I think that this is the biggest issue has Parse right now. With a good LDS, it'll be great.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment