Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

99 data product uploads #107

Closed
wants to merge 113 commits into from
Closed

99 data product uploads #107

wants to merge 113 commits into from

Conversation

ri-pandey
Copy link
Contributor

@ri-pandey ri-pandey commented Nov 8, 2023

Description

Please provide a brief description of the changes made in this PR.

Related Issue(s)

Closes #99

If applicable, please reference the issue(s) that this PR addresses. If the PR does not address any specific issue, you can remove this section.

Changes Made

List the main changes made in this PR. Be as specific as possible.

  • Added upload functionality, with a form for metadata selection. Uploads are done in 2MB chunks.
  • Added two levels of checksum validation (per chunk, and post-file-creation)
  • Added worker to merge uploaded chunks into files, and initiate the Integrated workflow.
  • Added script that monitors pending uploads, and retries to process them. Uploads which have not been in COMPLETE status for more than 24 hours

Checklist

Before submitting this PR, please make sure that:

  • Your code passes linting and coding style checks.
  • Documentation has been updated to reflect the changes.
  • You have reviewed your own code and resolved any merge conflicts.
  • You have requested a review from at least one team member.
  • Any relevant issue(s) have been linked to this PR.

Additional Information

Add any other information or context that may be relevant to this PR. This can include potential impacts, known issues, or future work related to this change.

added ability to upload file

added new page for data product upload

created file upload part of stepper

show warning before leaving route

show failed status for failed files

added retry ability for failed files

show icons for indicating file upload status

added progress circle

reformat

added File Type dropdown to Data Product Upload stepper

perform validation before moving to next tab in stepper

moved File Type dropdown into separate component

added Raw Data select to form; disabled AutoCompleteSelect's value being controlled from outside

resized dropdowns, changed button layout

added validation for Source Raw Data

added async validation for Data Product name

use ternary

store files within Data Product's parent directory

added worker to merge file chunks
trigger worker after upload

provide file details needed for merging to worker

prevent form submission if no files selected

WIP - File Type not being seeded; show File Type in Uploads history

corrected association; updated layout

validate field before next step

changed alert style

tested merging code

WIP - resume ability for failed uploads

got retry button to be disabled/enabled based on upload status

fixed runtime error

tested showing error on upload fail, then showing correct status on retry, etc.

added colors for new status values

kick off worker after all files are uploaded

validate checksums during upload

provide ability to re-upload files in case of runtime errors

rearranging variables

added role to page

added role to page

don't show type UPLOAD data products on Raw Data page

use correct HTTP method

persist file metadata

moved checksum evaluation into then chain

removed redundant async/await

update file log's and upload log's status after merge

removed test exception

finished implementing retry-merge feature

delete partially merged file in case of exception

changed upload path, dataset type

added file_type field to Data Products table

don't allow spaces in Data Product name

added upload path to config

WIP - testing
ri-pandey and others added 28 commits February 21, 2024 12:31
* retrieve about text from api

* WIP

* escape input

* switched to markdown-it

* fixed some styles

* use toast to show errors

* size; reset after Cancel

* made height fixed; use break-words

* changed modal size

* changed Edit button

* allow users/operators to read

* some UI changes

* escape input; allow for creation or updating

* use popover to indicate syntax

* UI changes

* added unauthenticated route

* cleanup

* changed routes; UI changes

* WIP

* force equal heights; made break-words work

* use labels optionally

* WIP

* changed .gitignore; seed data

* WIP - HTML line breaks missing

* fixed several styles

* fixed spacing; fonts

* use constant height across different screen sizes

* labels as props

* edited seeding code
@ri-pandey
Copy link
Contributor Author

  • [ ]

@ri-pandey ri-pandey closed this Apr 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement "Upload" feature
2 participants