Doc hub proposal

[pingsutw]

TL;DR

Design Doc

Database Schema

Task Table

Name	Type
...	...
descriptionID	uint32

Description Table

Name	Type
resource	string
project	string
domain	string
name	string
version	string
descriptionID	uint32
Digest	[]byte
ShortDescription	string
LongDescription	[]byte
link	string

Create a task/workflow description

we have added description_entity to taskSpec / workflowSpec, so we'll register a description entity when we register a task / workflow.

1. Check if we have the same description entity.

Select * from Description WHERE resource = task AND project = flytesnacks AND domain = development AND NAME = tf_wf AND version = xyz

2. If the same description entity doesn't exist, then insert a new row to the Description Table.
3. update descriptionID in the specific row in the task table.

Get task description

Use descriptionID in the task table to find the specific description entity

Select description.* LEFT JOIN task ON description.descriptionID = task.descriptionID AND description.project = task.project AND description.domain = task.domain AND description.name = task.name WHERE task.project = flytesnacks AND task.domain = development AND NAME = task.tf_wf AND task.version = a123456

List task description

Select description.* LEFT JOIN task ON description.descriptionID = task.descriptionID AND description.project = task.project AND description.domain = task.domain AND description.name = task.name WHERE task.project = flytesnacks AND task.domain = development AND NAME = task.tf_wf LIMIT 20

Get / List the Task

Added short description to task proto, so we'll get short description when we get / list the task.

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

Complete description

^^^

Tracking Issue

flyteorg/flyte#531

Follow-up issue

NA

[codecov]

Codecov Report

Merging #452 (8e1f17d) into master (ee8de0f) will increase coverage by 0.09%.
The diff coverage is 61.57%.

@@            Coverage Diff             @@
##           master     #452      +/-   ##
==========================================
+ Coverage   59.98%   60.07%   +0.09%     
==========================================
  Files         164      168       +4     
  Lines       14628    14997     +369     
==========================================
+ Hits         8774     9010     +236     
- Misses       5086     5196     +110     
- Partials      768      791      +23     

Flag	Coverage Δ
unittests	60.07% <61.57%> (+0.09%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/repositories/config/migrations.go	2.98% <0.00%> (-0.22%)	⬇️
pkg/repositories/gorm_repo.go	0.00% <0.00%> (ø)
pkg/repositories/gormimpl/common.go	62.85% <ø> (ø)
pkg/rpc/adminservice/base.go	3.53% <0.00%> (-0.07%)	⬇️
pkg/rpc/adminservice/description_entity.go	0.00% <0.00%> (ø)
pkg/repositories/gormimpl/task_repo.go	68.26% <33.33%> (-5.93%)	⬇️
pkg/manager/impl/task_manager.go	64.00% <50.00%> (-0.93%)	⬇️
pkg/repositories/gormimpl/workflow_repo.go	68.04% <53.84%> (-0.85%)	⬇️
pkg/manager/impl/workflow_manager.go	56.72% <54.54%> (-0.26%)	⬇️
...g/repositories/gormimpl/description_entity_repo.go	65.82% <65.82%> (ø)
... and 8 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[pingsutw]

cc @katrogan mind taking a look

[wild-endeavor]

so this will always create a record in the description_entities table, even if the task/wf has no comment. do we want this?

[katrogan]

Can we use gorm conventions to make this relational? https://gorm.io/docs/has_one.html

[wild-endeavor]

in an earlier version of this code, the long description was in S3 or a blob store of some kind right? it now will store everything in the db?

[katrogan]

What are the length restrictions on the long description?

[pingsutw]

4KB, flytekit will offload the long description if it exceeds 4KB when registering

[wild-endeavor]

and should that left join be a right join katrina?

[katrogan]

Because this is stored in the descriptions table, we'll have to do a left join now on every GetTask and GetWorkflow to populate the short description? Any reason not to store it inline?

[pingsutw]

updated, store it inline now

[katrogan]

you can use FirstOrCreate when you create the description entity to get its db id and avoid this lookup

[katrogan]

who calls this since you always Create as a side effect of workflow and task registration?

[pingsutw]

sorry, I forgot to remove this. removed it.

[pingsutw]

updating the tests

[pingsutw]

some updates:

• Add more tests
• Store short description in task and workflow table, so we don't to join description entity table to get the short description.
• Remove descriptionID from task and workflow table because we don't need to join description entity table.