feat(controller): build dataset in server #2497

goldenxinxing · 2023-07-11T11:15:32Z

Description

add new operation 'ds_build_and_upload' in entrypoint
add dataset build API and support GC the unused files in build record which is succeess
add oss signedUrl API
add log API(Online and offline)
optimise the job and pod eventHandler's dispatch logic(do different operation based on their respective label)

Modules

Checklist

run code format and lint check
add unit test
add necessary doc

codecov · 2023-07-12T02:16:32Z

Codecov Report

Merging #2497 (20f5a52) into main (076a8b2) will decrease coverage by 8.23%.
The diff coverage is 44.16%.

@@             Coverage Diff              @@
##               main    #2497      +/-   ##
============================================
- Coverage     91.24%   83.01%   -8.23%     
- Complexity        0     2646    +2646     
============================================
  Files           102      448     +346     
  Lines         11785    24326   +12541     
  Branches          0     1452    +1452     
============================================
+ Hits          10753    20194    +9441     
- Misses         1032     3476    +2444     
- Partials          0      656     +656

Flag	Coverage Δ
controller	`75.10% <44.16%> (?)`
standalone	`91.36% <ø> (+0.12%)`	⬆️
unittests	`91.36% <ø> (+0.12%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...i/starwhale/mlops/api/DatasetBuildLogWsServer.java	`0.00% <0.00%> (ø)`
...java/ai/starwhale/mlops/api/DatasetController.java	`76.68% <0.00%> (ø)`
.../ai/starwhale/mlops/api/FileStorageController.java	`0.00% <0.00%> (ø)`
...arwhale/mlops/configuration/RunTimeProperties.java	`100.00% <ø> (ø)`
...ataset/build/converter/BuildRecordVoConverter.java	`0.00% <0.00%> (ø)`
...domain/dataset/build/mapper/BuildRecordMapper.java	`0.00% <0.00%> (ø)`
...hale/mlops/domain/system/SystemSettingService.java	`67.74% <ø> (ø)`
.../starwhale/mlops/schedule/k8s/JobEventHandler.java	`63.70% <12.50%> (ø)`
...iguration/security/DatasetBuildTokenValidator.java	`23.52% <23.52%> (ø)`
...ps/domain/dataset/build/log/BuildLogCollector.java	`27.27% <27.27%> (ø)`
... and 11 more

... and 330 files with indirect coverage changes

anda-ren · 2023-07-14T06:59:27Z

maybe we could reuse the job-task framework, so that there is no bother to write log collector/status watcher and token validator

goldenxinxing · 2023-07-17T01:55:23Z

maybe we could reuse the job-task framework, so that there is no bother to write log collector/status watcher and token validator

Yes, but this may be need extract some common logics, like whether to sync the job to datastore\ which resourcePool to use(image and dataset may use private resource) ..etc. so i will use annother PR(#2509) to fix them.

tianweidut

LGTM

jialeicui · 2023-07-19T02:31:20Z

server/controller/src/main/java/ai/starwhale/mlops/domain/dataset/DatasetService.java

+        var buildings = buildRecordMapper.selectBuildingInOneProjectForUpdate(
+                    project.getId(), request.getDatasetName());


This line means we can apply two buildings with case-sensitive dataset names like mnist and Mnist?
And it seems that we do not allow this in Line 440:

if (!ds.getDatasetName().equalsIgnoreCase(request.getDatasetName())) {

yes, you are right, i will fix it

jialeicui · 2023-07-19T02:32:38Z

server/controller/src/main/java/ai/starwhale/mlops/domain/dataset/DatasetService.java

+        var res = buildRecordMapper.insert(entity) > 0;
+        if (res) {


Is it necessary to check the return value?

jialeicui · 2023-07-19T02:38:01Z

server/controller/src/main/java/ai/starwhale/mlops/domain/dataset/DatasetService.java

+            var dataset = datasetMapper.findByName(record.getDatasetName(), record.getProjectId(), false);
+            if (dataset != null) {
+                var version = datasetVersionMapper.findByLatest(dataset.getId());
+                if (version != null) {
+                    datasetVersionMapper.updateShared(version.getId(), true);
+                }
+            }


Make an issue to fix the race condition when the other user updates the dataset?

jialeicui · 2023-07-19T02:44:57Z

server/controller/src/main/java/ai/starwhale/mlops/domain/filestorage/FileStorageService.java

+        if (dataRootPath.endsWith("/")) {
+            this.dataRootPath = dataRootPath;
+        } else {
+            this.dataRootPath = dataRootPath + "/";
+        }


Check if the root path is short enough to make sure that the generatePathPrefix less than 255?

goldenxinxing self-assigned this Jul 12, 2023

goldenxinxing force-pushed the feat/build_dataset_in_server branch 2 times, most recently from 39349db to d359e70 Compare July 12, 2023 11:25

goldenxinxing changed the title ~~WIP feat(controller): build dataset in server~~ feat(controller): build dataset in server Jul 13, 2023

goldenxinxing added e2e feature ✨ new feature labels Jul 13, 2023

goldenxinxing marked this pull request as ready for review July 13, 2023 11:07

goldenxinxing force-pushed the feat/build_dataset_in_server branch from 2cc9f98 to f8cb6f4 Compare July 13, 2023 11:15

goldenxinxing added 21 commits July 14, 2023 11:18

init

34cecaa

add build record(doing)

3eb9a13

add build record(doing)

bcbf75f

add build record(doing)

42fb30b

add build record(doing)

2cc3909

add build record(doing)

3cf6957

add build record(doing)

c2fa677

add build record(doing)

179af77

add build record(doing)

5e442e8

modify entrypoint

9c8d704

build and log

639075f

add log ws ang api

38d73ec

fix

9e4e72c

fix

7153496

fix

4a09ff1

fix

163f9f6

fix

5753627

fix

ec7392b

fix

204b75d

fix

faab9b5

login in entrypoint

2576f7d

goldenxinxing added 6 commits July 14, 2023 11:26

tune file delete api

e074fc7

add ut

58c1e07

rename

4fb5f56

fix test

7de277b

fix multi pods

b1897d9

fix ut

73cbce7

goldenxinxing force-pushed the feat/build_dataset_in_server branch from bd1f1d4 to 73cbce7 Compare July 14, 2023 04:29

goldenxinxing requested review from tianweidut, xuchuan, jialeicui, anda-ren and waynelwz July 14, 2023 04:30

goldenxinxing added 3 commits July 14, 2023 14:13

fix gc

bb4baa4

fix log

905a512

fix log

4f82025

goldenxinxing added 3 commits July 14, 2023 16:21

fix online log

bea3169

check dataset name

600f1d7

check dataset name

b63ec87

goldenxinxing added 6 commits July 17, 2023 14:49

config resource

5a67179

download url with parallel

8c7291b

optimise log collector

d714992

optimise log collector

8171d16

optimise status

bc5542f

optimise list by status

20f5a52

tianweidut approved these changes Jul 19, 2023

View reviewed changes

tianweidut merged commit 7c7ff4f into star-whale:main Jul 19, 2023
42 checks passed

jialeicui reviewed Jul 19, 2023

View reviewed changes

goldenxinxing deleted the feat/build_dataset_in_server branch August 31, 2023 03:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(controller): build dataset in server #2497

feat(controller): build dataset in server #2497

goldenxinxing commented Jul 11, 2023 •

edited

Loading

codecov bot commented Jul 12, 2023 •

edited

Loading

anda-ren commented Jul 14, 2023

goldenxinxing commented Jul 17, 2023 •

edited

Loading

tianweidut left a comment

jialeicui Jul 19, 2023

goldenxinxing Jul 19, 2023

jialeicui Jul 19, 2023

jialeicui Jul 19, 2023

goldenxinxing Jul 19, 2023

jialeicui Jul 19, 2023

goldenxinxing Jul 19, 2023

		var buildings = buildRecordMapper.selectBuildingInOneProjectForUpdate(
		project.getId(), request.getDatasetName());

feat(controller): build dataset in server #2497

feat(controller): build dataset in server #2497

Conversation

goldenxinxing commented Jul 11, 2023 • edited Loading

Description

Modules

Checklist

codecov bot commented Jul 12, 2023 • edited Loading

Codecov Report

anda-ren commented Jul 14, 2023

goldenxinxing commented Jul 17, 2023 • edited Loading

tianweidut left a comment

Choose a reason for hiding this comment

jialeicui Jul 19, 2023

Choose a reason for hiding this comment

goldenxinxing Jul 19, 2023

Choose a reason for hiding this comment

jialeicui Jul 19, 2023

Choose a reason for hiding this comment

jialeicui Jul 19, 2023

Choose a reason for hiding this comment

goldenxinxing Jul 19, 2023

Choose a reason for hiding this comment

jialeicui Jul 19, 2023

Choose a reason for hiding this comment

goldenxinxing Jul 19, 2023

Choose a reason for hiding this comment

goldenxinxing commented Jul 11, 2023 •

edited

Loading

codecov bot commented Jul 12, 2023 •

edited

Loading

goldenxinxing commented Jul 17, 2023 •

edited

Loading