Add AllenNLP integration. #1086

himkt · 2020-04-03T14:18:55Z

This PR introduces AllenNLP integration to run HPO with AllenNLP configuration files.

Example

def objective(trial):
    trial.suggest_uniform('LEARNING_RATE', 1e-2, 1e-1)
    trial.suggest_uniform('DROPOUT', 0.0, 0.5)
    trial.suggest_int("MAX_FILTER_SIZE", 3, 6)
    trial.suggest_int("NUM_FILTERS", 16, 128)
    trial.suggest_int("NUM_OUTPUT_LAYERS", 1, 3)
    trial.suggest_int("HIDDEN_SIZE", 16, 128)

    serialization_dir = "test"
    executor = AllenNLPExecutor(
        trial,
        "classifier.jsonnet",
        serialization_dir,
        use_poetry=True
    )
    return executor.run()


if __name__ == "__main__":
    study = optuna.create_study(direction="maximize")
    study.optimize(objective, n_trials=50)

#1078

codecov-io · 2020-04-07T19:40:42Z

Codecov Report

Merging #1086 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1086   +/-   ##
=======================================
  Coverage   90.46%   90.46%           
=======================================
  Files         131      131           
  Lines       11331    11331           
=======================================
  Hits        10251    10251           
  Misses       1080     1080

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 928cefd...928cefd. Read the comment docs.

examples/allennlp/classifier.jsonnet

himkt · 2020-04-07T20:14:34Z

I think this PR is ready for review.
Could someone please take a look?

examples/allennlp/allennlp_jsonnet.py

examples/allennlp/classifier.jsonnet

optuna/integration/allennlp.py

tests/integration_tests/allennlp_tests/test_allennlp.py

examples/allennlp/allennlp_jsonnet.py

optuna/integration/allennlp.py

crcrpar

Thanks for adopting the new Allennlp api!

Why does the format of test.jsonnet look different from that of example.jsonnet?

toshihikoyanase

I have some small comments.

examples/allennlp/allennlp_jsonnet.py

toshihikoyanase · 2020-04-13T09:13:00Z

examples/allennlp/classifier.jsonnet

+  "numpy_seed": 42,
+    "pytorch_seed": 42,


numpy_seed and pytorch_seed seem to be in the same nest-level, but they have different indent-levels.

Thank you for pointing out, I run jsonnetfmt in d591f53.

toshihikoyanase · 2020-04-13T09:31:46Z

optuna/integration/allennlp.py

+        """Train a model using allennlp."""
+
+        for package_name in self._include_package:
+            allennlp.common.util.import_module_and_submodules(package_name)


I think any test cases do not visit this line. Please add test cases.

I added tests in 6ac976b and found there is a breaking change in this feature.

import_module_and_submodule will be introduced in the next release allennlp,
which is the renamed method of import_modules in v0.9.0.

toshihikoyanase · 2020-04-13T09:33:44Z

optuna/integration/allennlp.py

+        serialization_dir: str,
+        metrics: str = "best_validation_accuracy",
+        *,
+        include_package: List[str] = []


Please add a description of include_package to the docstring.

Added in 6ac976b.

himkt · 2020-04-13T14:56:23Z

@crcrpar Thank you for patient reviews!
I reformatted jsonnet files by jsonnetfmt. (d591f53)

himkt · 2020-04-13T17:00:25Z

27e4d49

AllenNLPExecutor now parses jsonnet itself and calls train_model instead of train_model_from_file.
In this change, an executor neither calls allennlp cli nor depends on environment variables,
which means that an executor can run in multiple threads/processes.

tests/integration_tests/allennlp_tests/test_allennlp.py

himkt

0e14a39

Comments for parsing jsonnet.
(FYI. @crcrpar @toshihikoyanase )

tests/integration_tests/allennlp_tests/tiny_single_id.py

optuna/integration/allennlp.py

himkt · 2020-04-15T12:42:58Z

optuna/integration/allennlp.py

+    TrackerCallback = object
+
+
+@experimental("1.4.0")


I made AllenNLPExecutor experimental since the major release of AllenNLP will come soon.
There are a lot of updates in their usages and we could face the necessity of changing the interface.
For more information, please see https://github.com/allenai/allennlp/releases/tag/v1.0-prerelease.

himkt · 2020-04-15T13:25:08Z

@toshihikoyanase
Thank you for your reviewing, I revised my PR including some modification and the changing support level of AllenNLPExecutor (normal -> experimental), please take a look again.

I also would like to express my gratitude to @crcrpar for patient helps.

Co-Authored-By: Masaki Kozuki <masaki.kozuki.2014@gmail.com>

🙀 Co-Authored-By: Masaki Kozuki <masaki.kozuki.2014@gmail.com>

Co-Authored-By: Masaki Kozuki <masaki.kozuki.2014@gmail.com>

toshihikoyanase

Thank you for your update. It basically looks good to me.
I added small comments about docstring. Please check them.

optuna/integration/allennlp.py

toshihikoyanase · 2020-04-20T07:05:23Z

optuna/integration/allennlp.py

+
+    This feature is experimental since AllenNLP major release will come soon.
+    The interface may change without prior notice to correspond to the update.
+


How about adding the link to the example as can be seen in TensorFlow's integration?

See the example if you want to add a pruning hook to TensorFlow’s estimator.

Thank you for the suggestion.
Added in 1533f6a.

Co-Authored-By: Toshihiko Yanase <toshihiko.yanase@gmail.com>

…a into feature/allennlp-cli-example

toshihikoyanase

LGTM. Thank you for your great contribution!

himkt commented Apr 7, 2020

View reviewed changes

examples/allennlp/classifier.jsonnet Outdated Show resolved Hide resolved

himkt marked this pull request as ready for review April 7, 2020 20:14

crcrpar self-requested a review April 8, 2020 02:23

crcrpar reviewed Apr 8, 2020

View reviewed changes

crcrpar reviewed Apr 9, 2020

View reviewed changes

examples/allennlp/allennlp_jsonnet.py Outdated Show resolved Hide resolved

crcrpar added feature Change that does not break compatibility, but affects the public interfaces. optuna.integration Related to the `optuna.integration` submodule. This is automatically labeled by github-actions. labels Apr 9, 2020

crcrpar self-requested a review April 9, 2020 20:54

toshihikoyanase self-assigned this Apr 10, 2020

crcrpar reviewed Apr 10, 2020

View reviewed changes

examples/allennlp/allennlp_jsonnet.py Show resolved Hide resolved

examples/allennlp/allennlp_jsonnet.py Outdated Show resolved Hide resolved

optuna/integration/allennlp.py Outdated Show resolved Hide resolved

himkt changed the title ~~Add AllenNLP cli integration~~ Add AllenNLP integration Apr 12, 2020

himkt changed the title ~~Add AllenNLP integration~~ Add AllenNLP integration. Apr 12, 2020

crcrpar reviewed Apr 12, 2020

View reviewed changes

toshihikoyanase requested changes Apr 13, 2020

View reviewed changes

himkt changed the title ~~Add AllenNLP integration.~~ [WIP] Add AllenNLP integration. Apr 13, 2020

himkt changed the title ~~[WIP] Add AllenNLP integration.~~ Add AllenNLP integration. Apr 13, 2020

crcrpar reviewed Apr 13, 2020

View reviewed changes

tests/integration_tests/allennlp_tests/test_allennlp.py Outdated Show resolved Hide resolved

himkt commented Apr 14, 2020

View reviewed changes

himkt mentioned this pull request Apr 14, 2020

[Proposal] Add allennlp integration #1012

Closed

crcrpar reviewed Apr 15, 2020

View reviewed changes

tests/integration_tests/allennlp_tests/tiny_single_id.py Outdated Show resolved Hide resolved

crcrpar requested a review from toshihikoyanase April 15, 2020 06:54

crcrpar reviewed Apr 15, 2020

View reviewed changes

optuna/integration/allennlp.py Show resolved Hide resolved

himkt commented Apr 15, 2020

View reviewed changes

himkt added 2 commits April 18, 2020 13:47

Add example using allennlp cli

7ffdd3d

Update command

f8e31e7

himkt and others added 18 commits April 18, 2020 13:47

Update tests to apply latest design change

87732f0

Fix importing of allennlp and update docstring

024a194

Add tests using include_package

450c11d

Run jsonnetfmt

7dc0cc4

Add reference to allennlp single_id_token_indexer

6e02e0e

Tweak fix

cfe4401

Update tests/integration_tests/allennlp_tests/test_allennlp.py

2362685

Co-Authored-By: Masaki Kozuki <masaki.kozuki.2014@gmail.com>

Add comments in _build_params for developers

bce019d

Tweak fix

dd5928f

Update tests/integration_tests/allennlp_tests/tiny_single_id.py

16a1ee3

🙀 Co-Authored-By: Masaki Kozuki <masaki.kozuki.2014@gmail.com>

Update optuna/integration/allennlp.py

61e92db

Co-Authored-By: Masaki Kozuki <masaki.kozuki.2014@gmail.com>

Fix docstring

9380394

Follow coding-style

44152ff

Tweak fix in example jsonnet

54c1d27

Add decorator experimental

dc139b9

Restrict allennlp version

9d07306

cli -> jsonnet

c9d25b8

Update path to allennlp examples

928cefd

crcrpar approved these changes Apr 18, 2020

View reviewed changes

toshihikoyanase reviewed Apr 20, 2020

View reviewed changes

himkt and others added 5 commits April 21, 2020 00:50

Support allennlp>0.9.0 experimentally

6789cc9

Update optuna/integration/allennlp.py

5d47b58

Co-Authored-By: Toshihiko Yanase <toshihiko.yanase@gmail.com>

Merge branch 'feature/allennlp-cli-example' of github.com:himkt/optun…

3ec29ba

…a into feature/allennlp-cli-example

Add link to example

1533f6a

Add minor comment for jsonnet

f794eb6

toshihikoyanase approved these changes Apr 21, 2020

View reviewed changes

toshihikoyanase added this to the v1.4.0 milestone Apr 21, 2020

toshihikoyanase merged commit 4f1b4e3 into optuna:master Apr 21, 2020

himkt deleted the feature/allennlp-cli-example branch April 21, 2020 00:59

toshihikoyanase mentioned this pull request Apr 21, 2020

[Proposal] Add integration with allennlp training with jsonnet config files. #1078

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add AllenNLP integration. #1086

Add AllenNLP integration. #1086

himkt commented Apr 3, 2020 •

edited

Loading

codecov-io commented Apr 7, 2020 •

edited

Loading

himkt commented Apr 7, 2020

crcrpar left a comment

toshihikoyanase left a comment

toshihikoyanase Apr 13, 2020

himkt Apr 13, 2020 •

edited

Loading

toshihikoyanase Apr 13, 2020

himkt Apr 13, 2020 •

edited

Loading

toshihikoyanase Apr 13, 2020

himkt Apr 13, 2020

himkt commented Apr 13, 2020 •

edited

Loading

himkt commented Apr 13, 2020

himkt left a comment •

edited by crcrpar

Loading

himkt Apr 15, 2020

himkt commented Apr 15, 2020

toshihikoyanase left a comment

toshihikoyanase Apr 20, 2020

himkt Apr 20, 2020

toshihikoyanase left a comment


		This feature is experimental since AllenNLP major release will come soon.
		The interface may change without prior notice to correspond to the update.

Add AllenNLP integration. #1086

Add AllenNLP integration. #1086

Conversation

himkt commented Apr 3, 2020 • edited Loading

Example

codecov-io commented Apr 7, 2020 • edited Loading

Codecov Report

himkt commented Apr 7, 2020

crcrpar left a comment

Choose a reason for hiding this comment

toshihikoyanase left a comment

Choose a reason for hiding this comment

toshihikoyanase Apr 13, 2020

Choose a reason for hiding this comment

himkt Apr 13, 2020 • edited Loading

Choose a reason for hiding this comment

toshihikoyanase Apr 13, 2020

Choose a reason for hiding this comment

himkt Apr 13, 2020 • edited Loading

Choose a reason for hiding this comment

toshihikoyanase Apr 13, 2020

Choose a reason for hiding this comment

himkt Apr 13, 2020

Choose a reason for hiding this comment

himkt commented Apr 13, 2020 • edited Loading

himkt commented Apr 13, 2020

himkt left a comment • edited by crcrpar Loading

Choose a reason for hiding this comment

himkt Apr 15, 2020

Choose a reason for hiding this comment

himkt commented Apr 15, 2020

toshihikoyanase left a comment

Choose a reason for hiding this comment

toshihikoyanase Apr 20, 2020

Choose a reason for hiding this comment

himkt Apr 20, 2020

Choose a reason for hiding this comment

toshihikoyanase left a comment

Choose a reason for hiding this comment

himkt commented Apr 3, 2020 •

edited

Loading

codecov-io commented Apr 7, 2020 •

edited

Loading

himkt Apr 13, 2020 •

edited

Loading

himkt Apr 13, 2020 •

edited

Loading

himkt commented Apr 13, 2020 •

edited

Loading

himkt left a comment •

edited by crcrpar

Loading