Epic: GREI - Dataverse Rearchitecture

[cmbz]

Overview

"Improve three areas of the repository UI: 1) facilitate discovery and exploration of data by making it easier to customize collections of datasets, identify types of data more clearly, expand and improve tools to explore data within repositories, 2) expand and clarify the dataset creation and publishing workflow to make it easier to add a wider variety of data with appropriate metadata, 3) extend the RESTful APIs in the Dataverse software to enable more easily separating the backend web services that support the core Dataverse functionality with the frontend, and build scalable and flexible open-source UI modules for the Harvard Dataverse repository, usable by other repositories." (Was NIH OTA Aim 7)

Try out SPA Beta here: https://beta.dataverse.org/spa

Tasks

2024 Q2

• See: Proposal: SPA Beta Features for Q2 2024
• Note: Epic completeness varies according to year, quarter, and goals
• Infinite Scrolling
• Creation of datasets
• Editing datasets metadata
• Basic file upload
• Basic creation of collections (nice to have)
• Publishing datasets (nice to have and candidate for community contribution)

2024 Q3

• See proposal: SPA Beta Features for Q3 2024
• Q3 planning in progress
• Collection page results of all types (Files, Datasets and Collections)
• Collection page basic filtering options and search
• Search home page (from the epic)
• Create Collection page (Full version)
• Enhanced File upload

2024 Q4

• Pending

Resources

• Project: NIH GREI - Generalist Repository Ecosystem Initiative #124
• Also, see the following project boards:
  • Global Backlog: Resarch SPA Beta and Rearch Auth
  • Sprint Board
  • Project Google Folder (includes weekly status reports)

[cmbz]

Status: January 2024 Update

Work continues towards accomplishing the Must Have SPA features (described in: Proposal: SPA Beta Features) in advance of the SPA Beta demo at the upcoming Dataverse Community Meeting, 2024 and the 6.2 release, scheduled for 2024/03/31. Progress includes a functional Home Page, File Page, and elements of the Dataset Creation Page. Several backend API extensions to support the SPA were also completed.

Completed

• Add displayOrder, typeClass and isRequired fields to DatasetFieldType payload dataverse#10225
• 10202 extend get version files api to include total file count dataverse#10215
• 263 - Add file metadata section to the File page dataverse-frontend#281
• 279 - Fix remove loading text dataverse-frontend#287
• 109 - Add totalFilesCount to GetDatasetFiles use case dataverse-client-javascript#116
• 253 - Fix do not show pagination if there is only 1 page + files results copy dataverse-frontend#277
• Feature/231 create dataset boilerplate dataverse-frontend#251
• 264 - Add the AccessFileMenu to File Page dataverse-frontend#286
• Add displayOrder, typeClass and isRequired fields to DatasetFieldType payload dataverse#10225
• 261 - Add File Citation to File Page dataverse-frontend#269
• 262 - File labels section of the File page dataverse-frontend#271

[cmbz]

Status: February 2024 Update

Work continues towards accomplishing the Must Have SPA features (described in: Proposal: SPA Beta Features) in advance of the SPA Beta demo at the upcoming Dataverse Community Meeting, 2024 in March, and the 6.2 release, scheduled for 2024/03/31. Progress includes: must-have improvements to the Home Page, File Page, and page navigation, as well nice-to-have improvements to the Dataset Creation Page. In addition, the team is developing a new SPA proposal for Q2 which will be made available for community comment.

Specific completed issues include:

• Create use case to create a dataset dataverse-client-javascript#110
• Modify Dataverse API calls to use 'excludeFiles' parameter dataverse-client-javascript#111
• Create new documentation for version 2.0.0 dataverse-client-javascript#113
• Create use case to get a file dataverse-client-javascript#115
• Create use case to get the file citation dataverse-client-javascript#117

Completed

dataverse

• Fix the current behavior when using the :latest tag in getVersion API calls for deaccessioned datasets and unauthenticated users dataverse#10207
• Create an endpoint to retrieve the file citation dataverse#10240
• Update get file endpoint to add a datasetVersion optional query parameter and extend its payload dataverse#10280
• Extend the API to retrieve owner info for files, datasets and collections dataverse#10286
• Update get file endpoint to add a datasetVersion optional query parameter and extend its payload dataverse#10299
• 10286 add owner info dataverse#10322
• file citation via API dataverse#10246
• 10207 bug api datasets versions latest fix dataverse#10212

dataverse-javascript-client

• Create use case to create a dataset dataverse-client-javascript#110
• Modify Dataverse API calls to use 'excludeFiles' parameter dataverse-client-javascript#111
• Create new documentation for version 2.0.0 dataverse-client-javascript#113
• Create use case to get a file dataverse-client-javascript#115
• Create use case to get the file citation dataverse-client-javascript#117
• Modify GetAllDatasetPreviews use case to accept collection id dataverse-client-javascript#121
• Extend Dataset and File models to contain owner information dataverse-client-javascript#122
• Extend GetFile use case to retrieve the dataset version information dataverse-client-javascript#123
• Modify getDataset repository API call to use 'excludeFiles' parameter dataverse-client-javascript#112
• Adds create dataset use case dataverse-client-javascript#114
• New documentation for js-dataverse v2 dataverse-client-javascript#118
• 115 - Create use case to get a file dataverse-client-javascript#119
• 121 - Modify GetAllDatasetPreviews use case to accept collectionId dataverse-client-javascript#124
• 117- Create use case to get the file citation dataverse-client-javascript#125
• 122 - Extend dataset and file models to contain owner information dataverse-client-javascript#127
• Add datasetVersionId support in getFile use case dataverse-client-javascript#128
• Adds returnDatasetVersion optional param to getFile use case dataverse-client-javascript#129

dataverse-frontend

• Review Chromatic current state  dataverse-frontend#270
• Move loading of File Permissions out of FilePermissionsProvider dataverse-frontend#272
• Use Query Param in Home Page dataset list pagination dataverse-frontend#280
• Display real file information in the File Page dataverse-frontend#283
• Complete UI Working Group Landing Page dataverse-frontend#288
• Dataverse Optimistic Lock Exception when calling DatasetHelper.destroyAll() dataverse-frontend#291
• Display real dataset version information in the File Page dataverse-frontend#295
• Add 'Add Data' button to the Home page and navbar dataverse-frontend#303
• Problem installing Design System package from npm dataverse-frontend#307
• Integrate breadcrumbs with new owner information in js-dataverse dataverse-frontend#311
• Integrate file citation text dataverse-frontend#312

[cmbz]

Status: March 2024 Update

• A demo of the new SPA design and implementation (Dataverse SPA Development) was presented at the Dataverse Community Meeting, 2024 in Texcoco, Mexico.
• Work continues towards next SPA milestones.

Completed

• Returning to author now requires a reason that is sent by email to the author dataverse#10137
• Add settings to make Tree View as a default option dataverse#8892
• Return to Author: When returning datasets that were submitted for review, I want to add a feedback note. dataverse#3702
• Feature/336 dynamically render form fields dataverse-frontend#357
• 326 - Developer guide - Writing test cases dataverse-frontend#349
• Feature/289 accessibility badge dataverse-frontend#346
• Feature/322 support infinite scrolling dataverse-frontend#342
• 313 - Release design system npm package version 1.1.0 dataverse-frontend#341
• 316 - Functional create dataset form dataverse-frontend#334
• add new pid settings, switch format dataverse-frontend#333
• fix docker compose file post #10357 merging dataverse-frontend#332
• 305 update chromatic configurations dataverse-frontend#325
• Create functional create dataset form with "hardcoded" citation metadata fields dataverse-frontend#316
• Create SPA development proposal for Q2 dataverse-frontend#314
• Release design system npm package version 1.2.0 dataverse-frontend#313
• Update Chromatic configurations dataverse-frontend#305
• Create Slides for DCM Presentation dataverse-frontend#302
• Create Demo Repository for DCM Presentation dataverse-frontend#301
• Prepare Workshop Presentation for Dataverse Community Meeting  dataverse-frontend#300
• Connect Create Dataset Form with js-dataverse use case dataverse-frontend#293
• Add accessibility testing badge to the README dataverse-frontend#289
• [Analysis] Add all required fields to the Create Dataset Form dataverse-frontend#282
• Add the AccessFile button to the File Page dataverse-frontend#264
• Add the FileLabels section to the File Page dataverse-frontend#262
• Add all required fields to the Create Dataset Form dataverse-frontend#252
• Add dataset info to the list of datasets in the Home Page dataverse-frontend#230

[cmbz]

Status: April 2024

Substantive progress has been made on the Collections and File Upload pages, improvements to pagination, and documentation. See below for specific PRs merged during April 2024.

Completed

dataverse repo

• Add isReleased field to GET dataverse API response dataverse#10448
• Extend the API to add the displayOnCreate flag in the DatasetFieldType and MetadataBlock payloads dataverse#10389
• Add latestVersionPublishingStatus field to dataset version payload dataverse#10330

dataverse-frontend

• Update CONTRIBUTING doc with "good first issue" and "help wanted" labels dataverse-frontend#382
• Feature/file upload page boilerplate dataverse-frontend#375
• Feature/358 add unpublished label to collections page dataverse-frontend#372
• Dynamically render form field in Create Dataset Form + Fields Validations dataverse-frontend#369
• Feature/338 form validation library dataverse-frontend#368
• Fix/273 deaccessioned dataset unauthenticated dataverse-frontend#367
• Refactor/276 avoid multiple api calls dataverse-frontend#362
• 315 - Developer guide: Architecture design methodology dataverse-frontend#361
• Feature/294 clear test datasets dataverse-frontend#360
• 339 add alerts for create dataset dataverse-frontend#354
• 26 - Add frontend guides dataverse-frontend#353
• 344 - Display real collection data in the Collection Page dataverse-frontend#352
• 337 multiple values field type dataverse-frontend#351

dataverse-client-javascript

• 140 transform html to markdown for Collection description dataverse-client-javascript#145
• Create/destroy datasets on each IT/functional run dataverse-client-javascript#143
• Get collection metadata blocks dataverse-client-javascript#141
• 133 - Add publish dataset use case dataverse-client-javascript#137

[cmbz]

Status: May 2024

See details here: Rearchitecture Front End Weekly Update 🔁 - May 30th

Several features are in progress, are in Q/A, or have been merged:

• Host Collection field to the Create Dataset Form (Merged)
• Multiple dynamic fields (Merged)
• Upload Files Page in progress
• Edit Dataset (next big Q2 feature) will re-use code from Create Dataset
• Edit Dataset use case has been developed and merged
• Create Dataset page: Match API Email validation (Q/A)
• Extend Breadcrumbs generator for action items (Merged)
• Collection page: Bug addressed for collection page datasets not being refetched after navigating from a sub collection (Ready for QA)
• Trigger github actions on PR (Merged)

Completed

dataverse

• Support sitemaps with more than 50,000 items dataverse#10321
• Extend Dataverse metadatablocks API to retrieve metadata fields configured as required in the General Information page dataverse#10498

dataverse-frontend

• feature/380 Extend Breadcrumbs generator for action items dataverse-frontend#396
• add PR triggers to github action workflows dataverse-frontend#401
• Feature/340 collection host input dataverse-frontend#399
• 347 Add eslint plugin react hooks dataverse-frontend#359
• Multiple dynamic fields - Create Dataset Form dataverse-frontend#395
• 387 Fix Race Condition in e2e tests dataverse-frontend#391
• 378 spa milestones doc dataverse-frontend#392
• Feature/355 select multiple component dataverse-frontend#385
• Feat: Add pagination total results dataverse-frontend#383
• fix: map jsDataset.id and persistentId in JSDatasetMapper.toHierarachy() dataverse-frontend#404

dataverse-client-javascript

• Edit dataset use case dataverse-client-javascript#151

[cmbz]

Status: June 2024

This month, the SPA saw the following accomplishments and progress:

• File Upload
  • File Upload use cases (Merged) Adds use cases for uploading files through direct upload API dataverse-client-javascript#153
  • File Upload UI (Merged) File Uploader dataverse-frontend#397
  • File Upload use cases (Ready for QA) Adds use cases for uploading files through direct upload API dataverse-client-javascript#153
  • File Upload Integration (Work In Progress): Integrate file upload page with js-dataverse use cases dataverse-frontend#417
• Dataset page
  • Infinite scrolling on dataset files (Merged) 405 Infinite Scrolling on Dataset Files dataverse-frontend#415
  • Storybook example
• Edit Dataset metadata page
  • Feat/406 add edit dataset metadata form dataverse-frontend#422 (Ready for Review)
• Create collection page
  • Basic use case for creating a collection (Ready for QA) Adds create collection use case dataverse-client-javascript#155
• Fix dataverses/{id}/metadatablocks endpoint
  • (In Review) Fix metadata field type display condition in dataverses/{id}/metadatablocks API endpoint dataverse#10642
• Create Dataset page
  • Prefill form fields with user info (Merged) 407 create dataset prefill dataverse-frontend#414
  • Match API Email validation (Merged) Fix/393 Match api email validation dataverse-frontend#402
• Q3 feature proposal doc ready for review
  • https://docs.google.com/document/d/1n13bBaYDpbs8TouXKlZfX07v2D4ZVfjabip4vu-kjEQ

[cmbz]

Status: Year 3, Q1

• Year 3 Q1 saw a number of SPA developments, including a demo of the new SPA design and implementation (Dataverse SPA Development) presented at the Dataverse Community Meeting, 2024 in Texcoco, Mexico. Substantive progress was made on the Collections and File Upload pages, such as improvements to pagination and documentation and infinite scrolling support. Additional accomplishments included enhancing the Create Dataset Form, support for multiple dynamic fields, and finalizing the Edit Dataset use case.
• See: NIH GREI Quarterly Progress Report: Year3 Q1 (March 2024 - May 2024)