Supporting the simple(st) user story

[rokroskar]

This is a meta-issue to keep track of work done to support the basic data science user story.

Please add a link to the relevant issue once it's created in the corresponding repository. Track the fine-grained issues in sprint project.

0. Housekeeping

• Building docs from development on readthedocs
  • renga-python
  • renga
• Updating all the READMEs on current masters to inform of work on development and make the instructions relevant to the development branch (see Development branches #116)
• Integrate pieces of old Makefile that builds images across all repos with the new Makefile in development (Integrate pieces of old Makefile that builds images across all repos with the new Makefile in development #121)
• Travis-ci integration test for the development branch of renga
• make make start should bring up all the needed services
  • ui (Configure Renga repository for new ui #125, related is Travis renku-ui#119))
  • gitlab
  • keycloak
  • storage (Integrate storage #128)

1. Initiate a project, import a dataset

• Finalize dataset import to work with renga repos (Import datasets from renku-aware repos renku-python#99)
• update .gitattributes to track outputs of renga run and datasets in LFS (provide a template LFS configuration for the newly initialized project renku-python#104)

• metadata
  • document the ontology for projects, datasets, and CWL steps/workflows (https://github.com/SwissDataScienceCenter/renga-ontology/issues/3)
  • include JSON-LD context with all metadata (include JSON-LD context with all metadata renku-python#105) (@jirikuncar)

2. Explore the dataset using a notebook (isolated environment)

• Look into capturing data artifacts the same way as with renga run

3. Run code on data, recording data provenance for reproducibility

• write user stories as test cases for renga run
• Fix bugs with provenance calculation (Fix bugs with provenance calculation renku-python#106)
• Resolve lineage in submodules (Resolve lineage in submodules renku-python#120)
• Implement renga status that checks the validity of data provenance ( Implement renga status that checks the validity of data provenance renku-python#107)
• Write out workflows, fix problem with relative paths (Construct workflows from steps  renku-python#98)

• (Dockerization & python environments -- for future sprint)

4. Push project to a server for sharing

• Finalize storage service
  • decouple from graph (decouple object storage backends from graph backend renku-storage#44)
  • update swagger and doc (update swagger renku-storage#50)
  • define bucket/backend for LFS (LFS object backend / bucket choice renku-storage#51)
• Memory provisioning for gitlab on testing.datascience.ch (@ebiiii / @leafty)
• Configure gitlab runners on k8s
  • Set up the ingress for notebook servers @leafty
  • Look into resource limitations of runners
• UI running on testing.datascience.ch
  • Resolve the token issue (reuse flask proxy from old UI? -> @3C111 / @leafty / @ebiiii) - authenticating the UI with keycloak done in resolves #105: renku-ui#118
  • Configure deployment
• Manipulate project access settings
• Initialize a project in UI (templates? Gitlab-ci, .renga metadata etc.)
• renga push/sync (index the provenance information)
• staging environment (https://github.com/SwissDataScienceCenter/incubator-renga-k8s/issues/1)

5. Validate results/workflows

• Troubleshoot the CI configuration -- related to renga runner rerun should fail gracefully if no tool is found renku-python#102

6. Initiate discussions (ku) on the work done

• create/participate in discussions (Enable contribution editing renku-ui#108)
  * Markdown editors
• File browser for linking to entities in the repo
• Connect notebooks to live notebook server (Notebook Running renku-ui#114)
  * Dataset view, data preview

[rokroskar]

closing -- open items moved to #135