[Extension] Add GluonTS extension #1903

Carkham · 2022-08-14T06:19:56Z

Description

We create GluonTS extension based on djl.

It contains three parts including: data entry, transform, translator.
In the data entry section, we provide efficient key-value pairs for data preprocessing and conversion into NDList in GluonTS datasets and translators. And in the inference task, we also designed a Forecast class similar to GluonTS, and provided related query methods.

In the transform part, the corresponding data pre-processing process is provided through GluonTSData, and the easy-to-use interface and static methods are provided to complete this process.
Among them, convert is responsible for the shape transformation of NDArray, feature is responsible for generating time series features, field is responsible for field modification, and split is responsible for sampling time series data in different time periods during inference and train.

In the translator section, we provide translators for two commonly used GluonTS time series models. Most of the content of these two Translators are similar, only some parameters have been modified. Please note that users can quickly build their own GluonTS model Translator according to the transform we provide.

At the same time, we also added two examples of inference with the GluonTS model in the examples.

See README.md for detailed description.

extensions/gluonTS/build.gradle

extensions/gluonTS/src/main/java/ai/djl/gluonTS/ForeCast.java

extensions/gluonTS/src/main/java/ai/djl/gluonTS/SampleForeCast.java

frankfliu · 2022-08-15T01:40:19Z

extensions/gluonTS/src/main/java/ai/djl/gluonTS/dataset/FieldName.java

+     *
+     * @return the String converted to lower case.
+     */
+    public String lowerCase() {


This function doesn't seem useful

Ok, I will delete it, it came from my first version

frankfliu · 2022-08-15T01:44:40Z

extensions/gluonTS/src/main/resources/log4j2.xml

@@ -0,0 +1,30 @@
+<?xml version="1.0" encoding="UTF-8"?>


we don't need log4j at lib level

frankfliu · 2022-08-15T01:45:09Z

extensions/gluonTS/src/main/java/ai/djl/gluonTS/examples/DeepARExample.java

@@ -0,0 +1,81 @@
+/*


Move examples into top level example folder

frankfliu · 2022-08-15T01:45:51Z

extensions/gluonTS/src/main/java/ai/djl/gluonTS/transform/Utils.java

@@ -0,0 +1,17 @@
+/*


Let's check in those class when needed

Please avoid use git merge in future, it's hard to read history

And I suggest you break your changes into smaller pieces if possible

Sorry for that, I wondering is there any tutorial or cmd for me to split the changes into pieces? Cuz, im not familiar with git

Sorry for that, I wondering is there any tutorial or cmd for me to split the changes into pieces? Cuz, im not familiar with git

Since everything is in your local and they are new files, you can create a new branch and manually copy these files and create a new PR with selected files. You can create multiple branches, each branch for a PR

Thanks! I got it.

…nown error occur. fix this using manager.create(new Shape(0))

…nction, remove log4j, move examples Co-authored-by: Frank Liu <frankfliu2000@gmail.com>

…ror case

frankfliu

Example still failing.
Please make sure output NDArrays are created with the same NDManager as input NDArray.

frankfliu · 2022-08-17T15:37:01Z

extensions/timeseries/src/main/java/ai/djl/timeseries/translator/DeepARTranslator.java

+    @Override
+    public NDList processInput(TranslatorContext ctx, TimeSeriesData input) {
+        NDManager manager = ctx.getNDManager();
+        this.startTime = input.getStartTime();


Use member variable startTime is not thread-safe.
You can pass input to processOutput() by set attachment.

ctx.setAttachment("input", input);

Because forecast in the example needs to access the model's predictor's output outside of the translator's ctx lifecycle, I new an NDManager in the constructor of the SampleForecast to create an NDArray to ensure that the result is not released. But this would make the NDArray lifecycle in SampleForecast inconvenient to manage, and I don't have a good solution for that. One idea is to replace NDArray with float[] in SampleForecast, but some of the statistical operations would be inconvenient.

@Carkham To clarify what you mentioned here, did you mean that you need to access the model predictor's output, which should not be released outside the translator's ctx?

codecov-commenter · 2022-08-17T15:52:31Z

Codecov Report

Merging #1903 (0cb0da3) into master (bb5073f) will decrease coverage by 2.11%.
The diff coverage is 67.24%.

@@             Coverage Diff              @@
##             master    #1903      +/-   ##
============================================
- Coverage     72.08%   69.97%   -2.12%     
- Complexity     5126     5791     +665     
============================================
  Files           473      569      +96     
  Lines         21970    25708    +3738     
  Branches       2351     2777     +426     
============================================
+ Hits          15838    17988    +2150     
- Misses         4925     6356    +1431     
- Partials       1207     1364     +157

Impacted Files	Coverage Δ
api/src/main/java/ai/djl/modality/cv/Image.java	`69.23% <ø> (-4.11%)`	⬇️
...rc/main/java/ai/djl/modality/cv/MultiBoxPrior.java	`76.00% <ø> (ø)`
...rc/main/java/ai/djl/modality/cv/output/Joints.java	`71.42% <ø> (ø)`
.../main/java/ai/djl/modality/cv/output/Landmark.java	`100.00% <ø> (ø)`
...main/java/ai/djl/modality/cv/output/Rectangle.java	`72.41% <0.00%> (ø)`
...i/djl/modality/cv/translator/BigGANTranslator.java	`21.42% <0.00%> (-5.24%)`	⬇️
...odality/cv/translator/BigGANTranslatorFactory.java	`33.33% <0.00%> (+8.33%)`	⬆️
...nslator/InstanceSegmentationTranslatorFactory.java	`14.28% <0.00%> (-3.90%)`	⬇️
.../cv/translator/SemanticSegmentationTranslator.java	`0.00% <0.00%> (ø)`
.../cv/translator/StyleTransferTranslatorFactory.java	`40.00% <ø> (ø)`
... and 473 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

…in README.md.

Carkham · 2022-08-24T03:20:36Z

Example still failing. Please make sure output NDArrays are created with the same NDManager as input NDArray.

Before modifying NDManager in this line, my local test can pass this example, but it fails after modification, and an error is reported as "Native resource has been release already". So I set the NDManager of SampleForecast.sample to the parent NDManager of the model output which actually is the NDManager of predicotr and it works well.

In this following example code segment:

try (ZooModel<TimeSeriesData, Forecast> model = criteria.loadModel();
        Predictor<TimeSeriesData, Forecast> predictor = model.newPredictor()) {
    TimeSeriesData input = new TimeSeriesData(1);
    input.setStartTime(LocalDateTime.parse("2011-01-29T00:00"));
    NDArray target =
            model.getNDManager()
                    .randomUniform(0f, 50f, new Shape(1857))
                    .toType(DataType.FLOAT32, false);
    input.setField(FieldName.TARGET, target);

    // This is SampleForecast containing NDArray sample and its NDManager is from the predictor so that it could not be released after predict().
    Forecast foreCast = predictor.predict(input); 

    return foreCast.mean().toFloatArray(); // Here I invoke samples.mean() to caculate the result.
}// Here predictor.close() is automatically called, and SampleForecast.sample is also released. I think handle like this would not create a memory leak

Change-Id: I3df5e17e427a0f21a1ded862d72e7c2a5ce894c7

KexinFeng · 2022-08-25T02:03:14Z

Before modifying NDManager in this line, my local test can pass this example, but it fails after modification, and an error is reported as "Native resource has been release already".

It looks like this error is not triggered in the PR test. If the error happens after the modification, would it be detected in the PR test?

* init commit of GluonTS, and the transform is organised by static method * debug only: add transformer example * feature: DeepARExample and model files * bug: when the start index >= end index in slice of the tsData, an unknown error occur. fix this using manager.create(new Shape(0)) * feature: extract interface from the static method in transform * fromat Java and checkStyle * review comments: rename module, delete default constructor, delete function, remove log4j, move examples Co-authored-by: Frank Liu <frankfliu2000@gmail.com> * delete the static NDManager and format the code. * Refactor classes in timeseries package * bug fix: add the batch_size dim for the Shape(0) data in the split error case * Rename ForeCast to Forecast. Add the description of module structure in README.md. * Replace the thread unsafe usage with ctx.getAttachment * Make predictor.manager to be the NDManager of SampleForecast.samples * Refactor timeseries API Change-Id: I3df5e17e427a0f21a1ded862d72e7c2a5ce894c7 Co-authored-by: Frank Liu <frankfliu2000@gmail.com>

Carkham requested review from zachgk and frankfliu as code owners August 14, 2022 06:19

Carkham changed the title ~~Gluon ts~~ [Extension] Add GluonTS extension Aug 14, 2022

frankfliu reviewed Aug 15, 2022

View reviewed changes

Carkham added 5 commits August 15, 2022 12:32

init commit of GluonTS, and the transform is organised by static method

ca91520

debug only: add transformer example

c5ab2c2

feature: DeepARExample and model files

3200946

bug: when the start index >= end index in slice of the tsData, an unk…

1d84817

…nown error occur. fix this using manager.create(new Shape(0))

feature: extract interface from the static method in transform

e400752

Carkham force-pushed the gluonTS branch from bfb2bd6 to 89589c4 Compare August 15, 2022 04:34

fromat Java and checkStyle

c5182ac

Carkham force-pushed the gluonTS branch 2 times, most recently from 9018e56 to 75a6d36 Compare August 15, 2022 11:48

review comments: rename module, delete default constructor, delete fu…

9a852ad

…nction, remove log4j, move examples Co-authored-by: Frank Liu <frankfliu2000@gmail.com>

Carkham force-pushed the gluonTS branch from 75a6d36 to 9a852ad Compare August 15, 2022 16:07

Carkham and others added 3 commits August 16, 2022 16:14

delete the static NDManager and format the code.

5e653c9

Refactor classes in timeseries package

b506da0

bug fix: add the batch_size dim for the Shape(0) data in the split er…

fdc9095

…ror case

frankfliu reviewed Aug 17, 2022

View reviewed changes

Carkham changed the title ~~[Extension] Add GluonTS extension~~ [WIP][Extension] Add GluonTS extension Aug 17, 2022

Carkham added 3 commits August 18, 2022 01:36

Rename ForeCast to Forecast. Add the description of module structure …

8ff1b8b

…in README.md.

Replace the thread unsafe usage with ctx.getAttachment

ef37282

Make predictor.manager to be the NDManager of SampleForecast.samples

2a62e47

Carkham changed the title ~~[WIP][Extension] Add GluonTS extension~~ [Extension] Add GluonTS extension Aug 18, 2022

frankfliu force-pushed the gluonTS branch from a832e0e to 64880de Compare August 24, 2022 19:33

Refactor timeseries API

0cb0da3

Change-Id: I3df5e17e427a0f21a1ded862d72e7c2a5ce894c7

frankfliu force-pushed the gluonTS branch from 64880de to 0cb0da3 Compare August 25, 2022 01:11

frankfliu approved these changes Aug 25, 2022

View reviewed changes

frankfliu merged commit 085741a into deepjavalibrary:master Aug 25, 2022

KexinFeng mentioned this pull request Aug 26, 2022

Will DJL support time series preprocess ? #1452

Closed

Carkham deleted the gluonTS branch September 15, 2022 09:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Extension] Add GluonTS extension #1903

[Extension] Add GluonTS extension #1903

Carkham commented Aug 14, 2022 •

edited by KexinFeng

frankfliu Aug 15, 2022

Carkham Aug 15, 2022

frankfliu Aug 15, 2022

Carkham Aug 15, 2022

frankfliu Aug 15, 2022

Carkham Aug 15, 2022

frankfliu Aug 15, 2022

frankfliu Aug 15, 2022

frankfliu Aug 15, 2022

Carkham Aug 15, 2022

frankfliu Aug 15, 2022

Carkham Aug 15, 2022

frankfliu left a comment

frankfliu Aug 17, 2022

Carkham Aug 17, 2022

KexinFeng Aug 25, 2022

codecov-commenter commented Aug 17, 2022 •

edited

Carkham commented Aug 24, 2022 •

edited

KexinFeng commented Aug 25, 2022

[Extension] Add GluonTS extension #1903

[Extension] Add GluonTS extension #1903

Conversation

Carkham commented Aug 14, 2022 • edited by KexinFeng

Description

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frankfliu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Aug 17, 2022 • edited

Codecov Report

Carkham commented Aug 24, 2022 • edited

KexinFeng commented Aug 25, 2022

Carkham commented Aug 14, 2022 •

edited by KexinFeng

codecov-commenter commented Aug 17, 2022 •

edited

Carkham commented Aug 24, 2022 •

edited