Optimize the time cost to go into training #410

rickycao-qy · 2020-08-05T11:31:48Z

This PR changes as following:

Data process and model training will do simultaneously
implement next and nextBatch function in dataloader which will return data once the data fetched has been processed by data-process

Now the plugin of data-process, tfjs-simplecnn, tjfs-train has been changed using this new way to train. The test pipeline is test/pieplines/mnist-image-classification.json

Baseline:
the baseline test pipeline is in this pr branch's test/pipelines/mnist-image-classification
The CPU used is 2.2 GHz Quad-Core Intel Core i7

version	time to go into training	whole process
current master branch	1 min 30 s	7-8 hour
older version(commit `a81e6a8`)	1min 15 s	15 min 25s
this PR	11 s	13 min

… into add/global-process

…global-process

…data-process-pipeline

packages/core/src/types/data/common.ts

FeelyChau · 2020-08-05T16:09:16Z

packages/plugins/data-access/coco-data-access/src/index.ts

@@ -40,6 +42,11 @@ class DataLoader implements ImageDataLoader {
      label: this.dataPairs[id].label
    };
  }
+
+  async setItem(id: number, sample: ImageSample) {


Suggested change

async setItem(id: number, sample: ImageSample) {

setItem(id: number, sample: ImageSample): void {

This function is extended from super-class async function. Acutally return value is Promise. I give it as async function since it's likely that in some cases we need to do some async logic inside setItem

If the return value is void, we need to use : Promise<void>.

The second part is the return type. We make it clear which is the return type by using an arrow (=>) between the parameters and the return type. As mentioned before, this is a required part of the function type, so if the function doesn’t return a value, you would use void instead of leaving it off.

https://www.typescriptlang.org/docs/handbook/functions.html#writing-the-function-type

FeelyChau · 2020-08-05T16:09:52Z

packages/plugins/data-access/csv-data-access/src/index.ts

@@ -32,6 +33,10 @@ class DataLoader implements CsvDataLoader {
  async getItem(id: number) {
    return this.records[id];
  }
+
+  async setItem(id: number, sample: CsvSample) {


Suggested change

async setItem(id: number, sample: CsvSample) {

setItem(id: number, sample: CsvSample): void {

packages/plugins/model-train/image-classification-tfjs-model-train/src/index.ts

run_pipeline.sh

…data-process-pipeline

yorkie · 2020-08-07T14:18:01Z

packages/core/src/types/data/common.ts

-  getItem: (id: number) => Promise<Sample>;
-  next?: () => Promise<Sample>;
-  nextBatch?: (batchSize: number) => Promise<Array<Sample>>;
+export abstract class DataLoader{


Suggested change

export abstract class DataLoader{

export abstract class DataLoader {

And a unit tests for this class is required :)

Lint problem has been fixed. I will add a unit test ASAP

Unit test has been added. Please have a check pls :)

yorkie · 2020-08-07T14:21:56Z

packages/costa/src/client/entry.ts

+      return;
+    }
+
+    if (pkg.pipcook.category === 'datasetProcess') {


Shall we move the datasetProcess to default handler?

Sure, have remove this

yorkie · 2020-08-07T14:25:09Z

@rickycao-qy @WenheLI Awesome works and reviewed some small nits, BTW there might be a little bit conflicts with #405, @FeelyChau how can we merge them?

yorkie

LGTM if CI is green

packages/plugins/data-process/tfjs-image-classification-process/tsconfig.json

FeelyChau

LGTM if CI passed.

WenheLI and others added 16 commits July 28, 2020 14:25

feat: add GlobalDataProcess plugin

772dd5a

fix: fix lint issue

9132604

fix: fix lint issue & rename datasetProcess

00539da

Merge branch 'add/global-process' of https://github.com/alibaba/pipcook…

523a24a

… into add/global-process

add

f72eae9

fix: add inline comments

ec6a3b5

Merge branch 'master' of https://github.com/alibaba/pipcook into add/…

50dff0b

…global-process

Merge branch 'master' of https://github.com/alibaba/pipcook into add/…

fc941ef

…global-process

run pipeline

5998960

Merge branch 'master' of https://github.com/alibaba/pipcook into add/…

6224db4

…global-process

Merge branch 'master' of https://github.com/alibaba/pipcook into add/…

250f6d0

…global-process

save

5b6165b

Merge branch 'master' of https://github.com/alibaba/pipcook into add/…

40385c8

…global-process

change plugin accordingly

7b3fbc8

Merge branch 'master' of https://github.com/alibaba/pipcook into add/…

a8ab4b8

…data-process-pipeline

delete uncessary log

e5ab252

rickycao-qy requested review from yorkie and WenheLI August 5, 2020 11:31

fix lint

6c9676d

WenheLI reviewed Aug 5, 2020

View reviewed changes

packages/core/src/types/data/common.ts Outdated Show resolved Hide resolved

packages/core/src/types/data/common.ts Outdated Show resolved Hide resolved

FeelyChau requested changes Aug 5, 2020

View reviewed changes

rickycao-qy added 5 commits August 6, 2020 10:31

fix

c7c8db5

change way how image-class tensorflow owrks

1775ae2

change resize to reasonable size

aa7803a

Merge branch 'master' of https://github.com/alibaba/pipcook into add/…

a529104

…data-process-pipeline

fix

e3ee6c7

yorkie reviewed Aug 7, 2020

View reviewed changes

move datasetprocess to default

960b1c6

test dataloader

48a2be7

yorkie previously approved these changes Aug 10, 2020

View reviewed changes

FeelyChau requested changes Aug 10, 2020

View reviewed changes

packages/plugins/data-process/tfjs-image-classification-process/tsconfig.json Outdated Show resolved Hide resolved

add eof

2641855

rickycao-qy dismissed yorkie’s stale review via 2641855 August 10, 2020 06:36

FeelyChau previously approved these changes Aug 10, 2020

View reviewed changes

fix api for python tf

c960e1b

rickycao-qy dismissed FeelyChau’s stale review via c960e1b August 10, 2020 07:10

yorkie approved these changes Aug 10, 2020

View reviewed changes

FeelyChau merged commit 1e183f2 into master Aug 10, 2020

FeelyChau deleted the add/data-process-pipeline branch August 10, 2020 09:06

WenheLI mentioned this pull request Sep 8, 2020

Stream-based Pipeline & Plugins #571

Closed

1 task

gindis pushed a commit to gindis/pipcook that referenced this pull request Sep 11, 2020

Optimize the time cost to go into training (alibaba#410)

fd4041d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimize the time cost to go into training #410

Optimize the time cost to go into training #410

rickycao-qy commented Aug 5, 2020 •

edited

FeelyChau Aug 5, 2020

rickycao-qy Aug 6, 2020

FeelyChau Aug 6, 2020

FeelyChau Aug 5, 2020

rickycao-qy Aug 7, 2020

yorkie Aug 7, 2020

yorkie Aug 7, 2020

rickycao-qy Aug 8, 2020

rickycao-qy Aug 10, 2020

yorkie Aug 7, 2020

rickycao-qy Aug 8, 2020

yorkie commented Aug 7, 2020

yorkie left a comment

FeelyChau left a comment

	async setItem(id: number, sample: ImageSample) {
	setItem(id: number, sample: ImageSample): void {

	async setItem(id: number, sample: CsvSample) {
	setItem(id: number, sample: CsvSample): void {

	export abstract class DataLoader{
	export abstract class DataLoader {

Optimize the time cost to go into training #410

Optimize the time cost to go into training #410

Conversation

rickycao-qy commented Aug 5, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yorkie commented Aug 7, 2020

yorkie left a comment

Choose a reason for hiding this comment

FeelyChau left a comment

Choose a reason for hiding this comment

rickycao-qy commented Aug 5, 2020 •

edited