Skip to content

Make objectId lexographically sortable #7830

Open
@JasCodes

Description

@JasCodes

New Feature / Enhancement Checklist

  • I am not disclosing a vulnerability.
    I am not just asking a question.
    I have searched through existing issues.

Current Limitation

objectID is not sortable by creation time.

Feature / Enhancement Description

Implement ULID spec based on https://www.npmjs.com/package/ulid-workers
One also don't need to store createdAt separately anymore and objectId can be sorted based on creation time without conflicts that might arise when you clone multiple objects from dashboard.

Example Use Case

I wanted to implement cursor based pagination. I wanted to say less than objectID with x limit to load next batch of items.

Activity

parse-github-assistant

parse-github-assistant commented on Mar 1, 2022

@parse-github-assistant

Thanks for opening this issue!

  • ❌ Please edit your post and use the provided template when creating a new issue. This helps everyone to understand your post better and asks for essential information to quicker review the issue.
JasCodes

JasCodes commented on Mar 1, 2022

@JasCodes
Author
davimacedo

davimacedo commented on Mar 1, 2022

@davimacedo
Member

The sortable object id by creation time is something that can be discussed but it is not something necessary for your use-case. You can implement cursor based pagination with the current objectId.

@mtrezza thoughts about this?

JasCodes

JasCodes commented on Mar 2, 2022

@JasCodes
Author

@davimacedo Thanks for coming back on this man. You can implement cursor based pagination with the current objectId. - Can you explain or refer me to doc/article for this? The column needs to be uniquely sortable no?

And is there way to overide objectId generation function on server setup?

davimacedo

davimacedo commented on Mar 2, 2022

@davimacedo
Member

The objectId is uniquely sortable. It does not represent the creation time, but it is sortable and it is unique. If you order the objects by objectId then you can use the objectId as the cursor key for the pagination. If you want to paginate according to the creation date, it is possible as well. You can combine (createdAt, objectId) and sort by the two columns, in this order, then use the combination of the two columns as the key for the cursor pagination.

Parse Server also supports custom object id: parse-community/Parse-SDK-JS#1309

JasCodes

JasCodes commented on Mar 2, 2022

@JasCodes
Author

You can combine (createdAt, objectId) and sort by the two columns not very performant unless I put a compound index on those in MongoDB yeah?
Parse Server also supports custom object id: https://github.com/parse-community/Parse-SDK-JS/pull/1309 But can I do it from server-side and not allow clients to set object id?
@davimacedo

JasCodes

JasCodes commented on Mar 2, 2022

@JasCodes
Author

You can combine (createdAt, objectId) and sort by the two columns, in this order, then use the combination of the two columns as the key for the cursor pagination. Do you mean to create another column for this?

davimacedo

davimacedo commented on Mar 2, 2022

@davimacedo
Member

You can combine (createdAt, objectId) and sort by the two columns not very performant unless I put a compound index on those in MongoDB yeah?

Yes. An index is welcome for this case.

Parse Server also supports custom object id: parse-community/Parse-SDK-JS#1309 But can I do it from server-side and not allow clients to set object id?

I've never tried that but I believe you can do that on a beforeSave trigger.

You can combine (createdAt, objectId) and sort by the two columns, in this order, then use the combination of the two columns as the key for the cursor pagination. Do you mean to create another column for this?

No. You don't need a new column. Your cursor will be the createdAt and objectId of the last object of the last page. You can pack them: concatenate them (and then optionally encode), or stringify an object with both of them (and then optionally encode), or any other mechanism. When retrieving the next page, you will unpack the last createdAt and last objectId. Then your query will sort the objects by createdAt and objectId at the same time in this order, and then filter by createdAt > lastCreatedAt or createdAt = lastCreatedAt && objectId > lastObjectId.

JasCodes

JasCodes commented on Mar 2, 2022

@JasCodes
Author

Thanks very much for getting back bro!! @davimacedo
createdAt > lastCreatedAt or createdAt = lastCreatedAt && objectId > lastObjectId I already tried that. Will compound index will be useful in this case? I was sorting them first, that's why I thought compound index. But this sounds like hit to performance for no reason.

Anyways it's far easier and I think performant to create a new column and generate ULID using beforeSave. And custom object id doesn't work on server-side btw as id and createdAt is assigned after beforeSave is run, I think.

But these customizations of setting objectKey or createdAt should be exposed on server configurations in triggers or just passing function in options. What do you think Mr. @mtrezza ?

JasCodes

JasCodes commented on Mar 2, 2022

@JasCodes
Author

@mtrezza I would even recommend using ULID as default id and createdAt can be just read from it. Infact cursor based pagination should be first-class citizen.

added
type:featureNew feature or improvement of existing feature
on Mar 3, 2022
mtrezza

mtrezza commented on Mar 3, 2022

@mtrezza
Member

Yes, I think allowing to set a custom object ID or overriding createdAt and any of the auto-generated fields server-side would be a useful feature, if that is not possible yet. Please feel free to look into that and submit a PR. There may be some hidden complexity depending on the field, as a fields may be used - in an opaque way - to infer object states.

ULID sounds like a good idea, but it may be a significant change for existing deployments, given the difference in characteristics to the currently used ID. Making ULID a default would certainly be possible, but I think only in the context of a Parse Server option that allows to determine the object ID server side. That new option could easily allow some pre-set ID generators and also allow to set a custom generator function.

You could go about it like this: find out where the object ID is currently being generated in Parse Server and expose that as a Parse Server option with some easy to configure pre-sets.

JasCodes

JasCodes commented on Mar 3, 2022

@JasCodes
Author

@mtrezza For sure, but I am really new to parse and unfamiliar to the codebase. Though I will love to be a regular code contributor as started using it with two commercial projects. And its always good to give back. Can somebody from the community who knows the codebase do a quick call with me? Is that possible? Are there any web meetups etc?

mtrezza

mtrezza commented on Mar 3, 2022

@mtrezza
Member

I suggest you look at the respective code path when saving an object (POST route). Then you can see where the object ID is assigned. That is the functionality you'd want to extend. In the contribution guide you can see how to introduce a new Parse Server option. Any questions, feel free to ask.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type:featureNew feature or improvement of existing feature

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Participants

      @JasCodes@davimacedo@mtrezza

      Issue actions

        Make objectId lexographically sortable · Issue #7830 · parse-community/parse-server