Skip to content

Allow Parse Job to be launched with more than 16Mb of data in params #8341

Closed
@codeKonami

Description

@codeKonami
Contributor

New Feature / Enhancement Checklist

  • I am not disclosing a vulnerability.
    I am not just asking a question.
    I have searched through existing issues.

Current Limitation

When running a Parse Job, the params is saved in the _JobStatus collection by default in MongoDB thus limiting the size of the params to the MongoDB BSON limit of 16Mb : https://www.mongodb.com/docs/manual/reference/limits/#:~:text=The%20maximum%20BSON%20document%20size,MongoDB%20provides%20the%20GridFS%20API.

Feature / Enhancement Description

We would need to modify this code :

return jobHandler.setRunning(jobName, params).then(jobStatus => {

Two ways of doing it :

  • We could pass a context while launching the Parse Job telling that we should not save the params.
  • We could measure the size of the params and therefore truncate the params inside the JobStatus document.

Example Use Case

For example I want to launch a job with 32Mb of data to analyse, I will launch the job but explicitly telling Parse to not save the params in database.

Alternatives / Workarounds

No easy workaround

3rd Party References

POSTing more than 16Mb of data is quite common everywhere :)

Activity

parse-github-assistant

parse-github-assistant commented on Nov 30, 2022

@parse-github-assistant

Thanks for opening this issue!

  • 🎉 We are excited about your ideas for improvement!
mtrezza

mtrezza commented on Nov 30, 2022

@mtrezza
Member

_JobStatus is an internal collection and therefore not supposed to be used for anything else than supporting features of Parse Server. Any change to that collection would not constitute a breaking change, so one can not rely on it, what is logged, how it's logged, or whether anything is logged at all.

In that context, the params stored cannot have any operational value, they could have at most analytical or troubleshooting value. In addition, storing operational data in an internal collection is not supposed to happen and we should probably implement a guideline for reviewers about that. Operational data may be sensitive, require data protection considerations, auditing, or other things that are problematic for internal collections.

My suggestion is to remove param logging. If params are currently used by Parse Server, then we could think about finding a replacement concept. Any logging of params can be implemented fine-grained and according to use case by the developer in the job script itself.

codeKonami

codeKonami commented on Dec 1, 2022

@codeKonami
ContributorAuthor

@mtrezza So if I understand you correctly, you think we should remove the params altogether?

mtrezza

mtrezza commented on Dec 2, 2022

@mtrezza
Member

I would say so, unless there is a good reason why they are logged. Did you look at the code to see whether they are used anywhere?

added a commit that references this issue on Dec 2, 2022
25e7938
codeKonami

codeKonami commented on Dec 2, 2022

@codeKonami
ContributorAuthor

The only good reason to store the params in the JobStatus document would be to have a retry mechanism of some sort. But as of today these params does not seem to be used anywhere.

mtrezza

mtrezza commented on Dec 2, 2022

@mtrezza
Member

Good to know that they aren't used. So I'd suggest to remove then completely.

danielsanfr

danielsanfr commented on Dec 8, 2022

@danielsanfr

If I understood the discussion correctly, the solution would be to keep this file in a separate location like a ParseFile, for example. So it would only be necessary to pass an objectId as a Job parameter.

Although today the parameters are not shown on the Job Status page, I would prefer not to remove them, as simply removing them does not improve privacy or security at all.

I've also thought about the possibility of making a retry mechanism, but I never had the time.
So, if you still want to remove something... Please, just remove it from the collection, because I have several Jobs using parameters and I believe that there are many more users who also use them.

Please disregard the comment, if I understood something wrong or if I didn't express myself well, because English is not my main language.

codeKonami

codeKonami commented on Dec 8, 2022

@codeKonami
ContributorAuthor

If I understood the discussion correctly, the solution would be to keep this file in a separate location like a ParseFile, for example. So it would only be necessary to pass an objectId as a Job parameter.

Although today the parameters are not shown on the Job Status page, I would prefer not to remove them, as simply removing them does not improve privacy or security at all.

I've also thought about the possibility of making a retry mechanism, but I never had the time. So, if you still want to remove something... Please, just remove it from the collection, because I have several Jobs using parameters and I believe that there are many more users who also use them.

Please disregard the comment, if I understood something wrong or if I didn't express myself well, because English is not my main language.

The params are only removed from the collection, they are still passed through the request to the Parse.Job function so no worries about that.

mtrezza

mtrezza commented on Dec 8, 2022

@mtrezza
Member

@danielsanfr did that answer address your concerns?

added a commit that references this issue on May 8, 2025
6340286
parseplatformorg

parseplatformorg commented on May 14, 2025

@parseplatformorg
Contributor

🎉 This change has been released in version 8.2.1-alpha.2

parseplatformorg

parseplatformorg commented on Jun 1, 2025

@parseplatformorg
Contributor

🎉 This change has been released in version 8.2.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:releasedReleased as stable versionstate:released-alphaReleased as alpha versiontype:featureNew feature or improvement of existing feature

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Participants

      @codeKonami@danielsanfr@mtrezza@parseplatformorg

      Issue actions

        Allow Parse Job to be launched with more than 16Mb of data in params · Issue #8341 · parse-community/parse-server