Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add export all rows of a class and export in JSON format #2361

Merged
merged 13 commits into from
Jan 25, 2023

Conversation

dblythy
Copy link
Member

@dblythy dblythy commented Jan 19, 2023

New Pull Request Checklist

Issue Description

It's currently only possible to export 10,000 rows, and only as a CSV

Closes: #1961

Approach

Uses Parse.Query.eachBatch to paginate over full class and exports as files when:

  • Array of objects is over 1gb in memory
  • New objects length is less than batch size (eachBatch has finished)

Screenshot 2023-01-19 at 3 45 34 pm

TODOs before merging

  • Add tests
  • Add changes to documentation (guides, repository pages, in-code descriptions)

@parse-github-assistant
Copy link

parse-github-assistant bot commented Jan 19, 2023

Thanks for opening this pull request!

  • 🎉 We are excited about your hands-on contribution!

@dblythy dblythy requested a review from a team January 19, 2023 05:40
mtrezza
mtrezza previously approved these changes Jan 20, 2023
Copy link
Member

@mtrezza mtrezza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@dblythy
Copy link
Member Author

dblythy commented Jan 22, 2023

Screen Shot 2023-01-23 at 10 19 15 am

@dblythy dblythy requested a review from a team January 22, 2023 23:29
Copy link
Member

@mtrezza mtrezza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice rework of the dialog!

I looked at the delete rows dialog, to use the same wording here:

Confirm this action
Enter "export all" to continue.

Unlike a delete action, it may not be clear why this is a critical operation. So in addition we could add a text that says:

⚠️ Exporting all rows may significantly impact resources.

Maybe you could add that below the "Estimated data size", or if it looks acceptable add it to the confirmation segment:

Confirm this action
⚠️ Exporting all rows may significantly impact resources. Enter "export all" to continue.

@dblythy
Copy link
Member Author

dblythy commented Jan 24, 2023

Screenshot 2023-01-25 at 10 51 31 am

How's this look?

Copy link
Member

@mtrezza mtrezza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@mtrezza mtrezza changed the title feat: Allow export all columns, allow export as JSON feat: Add export all rows of a class and export in JSON format Jan 25, 2023
@mtrezza mtrezza merged commit 9eb36a1 into parse-community:alpha Jan 25, 2023
@dblythy dblythy deleted the download-all branch January 25, 2023 13:22
@dblythy
Copy link
Member Author

dblythy commented Jan 25, 2023

Thanks for your help!

parseplatformorg pushed a commit that referenced this pull request Jan 25, 2023
# [5.1.0-alpha.5](5.1.0-alpha.4...5.1.0-alpha.5) (2023-01-25)

### Features

* Add export all rows of a class and export in JSON format ([#2361](#2361)) ([9eb36a1](9eb36a1))
@parseplatformorg
Copy link
Contributor

🎉 This change has been released in version 5.1.0-alpha.5

@parseplatformorg parseplatformorg added the state:released-alpha Released as alpha version label Jan 25, 2023
parseplatformorg pushed a commit that referenced this pull request Mar 1, 2023
# [5.1.0-beta.2](5.1.0-beta.1...5.1.0-beta.2) (2023-03-01)

### Bug Fixes

* Add dashboard option `cookieSessionMaxAge` to keep user logged in across browser sessions ([#2366](#2366)) ([9ea95fc](9ea95fc))
* Blank screen shown if server is unreachable; unsupported pages are accessible via direct URLs ([#2363](#2363)) ([9855258](9855258))
* Dashboard may display blank page when selecting an app after login ([#2375](#2375)) ([f399b91](f399b91))
* Data browser dialog "No data to display" may be outside of visible area in Safari browser ([#2387](#2387)) ([52bba62](52bba62))
* Internal error message on login with missing credential ([#2370](#2370)) ([9a6a31f](9a6a31f))
* Navigation to page fails if user re-login is required ([#2369](#2369)) ([0db6f55](0db6f55))
* Screen goes blank when trying to add column of type `Object` or `GeoPoint` ([#2384](#2384)) ([0886386](0886386))
* Text selection not visible in modal dialog header ([#2340](#2340)) ([fb0e79c](fb0e79c))

### Features

* Add export all rows of a class and export in JSON format ([#2361](#2361)) ([9eb36a1](9eb36a1))
* Add schema export ([#2362](#2362)) ([33df049](33df049))
@parseplatformorg
Copy link
Contributor

🎉 This change has been released in version 5.1.0-beta.2

@parseplatformorg parseplatformorg added the state:released-beta Released as beta version label Mar 1, 2023
parseplatformorg pushed a commit that referenced this pull request May 1, 2023
# [5.1.0](5.0.0...5.1.0) (2023-05-01)

### Bug Fixes

* Add dashboard option `cookieSessionMaxAge` to keep user logged in across browser sessions ([#2366](#2366)) ([9ea95fc](9ea95fc))
* Blank screen shown if server is unreachable; unsupported pages are accessible via direct URLs ([#2363](#2363)) ([9855258](9855258))
* Dashboard may display blank page when selecting an app after login ([#2375](#2375)) ([f399b91](f399b91))
* Data browser dialog "No data to display" may be outside of visible area in Safari browser ([#2387](#2387)) ([52bba62](52bba62))
* Internal error message on login with missing credential ([#2370](#2370)) ([9a6a31f](9a6a31f))
* Navigation to page fails if user re-login is required ([#2369](#2369)) ([0db6f55](0db6f55))
* Screen goes blank when trying to add column of type `Object` or `GeoPoint` ([#2384](#2384)) ([0886386](0886386))
* Text selection not visible in modal dialog header ([#2340](#2340)) ([fb0e79c](fb0e79c))

### Features

* Add export all rows of a class and export in JSON format ([#2361](#2361)) ([9eb36a1](9eb36a1))
* Add schema export ([#2362](#2362)) ([33df049](33df049))
* remove limitation to refresh Cloud Jobs list only after 30 seconds ([#2332](#2332)) ([ad1132f](ad1132f))
@parseplatformorg
Copy link
Contributor

🎉 This change has been released in version 5.1.0

@parseplatformorg parseplatformorg added the state:released Released as stable version label May 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
state:released Released as stable version state:released-alpha Released as alpha version state:released-beta Released as beta version
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Can't export more than 10k rows as csv
3 participants