New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PUBDEV-4271] H2O Modularization #915

Merged
merged 2 commits into from May 26, 2017

Conversation

Projects
None yet
4 participants
@mmalohlava
Member

mmalohlava commented Mar 15, 2017

This is WIP !

@vpatryshev

Kind of weird that there are 0 tests here.

@mmalohlava

This comment has been minimized.

Show comment
Hide comment
@mmalohlava

mmalohlava Mar 15, 2017

Member

@vpatryshev this is WIP, i just would like too see status of jenkins run.

Member

mmalohlava commented Mar 15, 2017

@vpatryshev this is WIP, i just would like too see status of jenkins run.

@jakubhava

This comment has been minimized.

Show comment
Hide comment
@jakubhava

jakubhava May 22, 2017

Contributor

Just so we don't forget about it. The capabilities rest endpoint is implemented in https://github.com/h2oai/h2o-3/tree/jh/exp/modularization-new and the commits can be cherry-picked from there

Contributor

jakubhava commented May 22, 2017

Just so we don't forget about it. The capabilities rest endpoint is implemented in https://github.com/h2oai/h2o-3/tree/jh/exp/modularization-new and the commits can be cherry-picked from there

@mmalohlava

This comment has been minimized.

Show comment
Hide comment
@mmalohlava

mmalohlava May 22, 2017

Member

@jakubhava thanks! I put it into the branch.

Member

mmalohlava commented May 22, 2017

@jakubhava thanks! I put it into the branch.

@mmalohlava mmalohlava changed the title from WIP: on the way of modularization to [PUBDEV-4271] H2O Modularization May 24, 2017

@mmalohlava

This comment has been minimized.

Show comment
Hide comment
@mmalohlava

mmalohlava May 25, 2017

Member

@h2o-ops please test!

Member

mmalohlava commented May 25, 2017

@h2o-ops please test!

@mmalohlava mmalohlava removed the WIP label May 25, 2017

@mmalohlava mmalohlava requested a review from michalkurka May 25, 2017

@@ -834,40 +842,40 @@ public static void registerRestApis(String relativeResourcePath) {
}
// Log extension registrations here so the message is grouped in the right spot.
for (AbstractH2OExtension e : H2O.getExtensions()) {

This comment has been minimized.

@michalkurka

michalkurka May 25, 2017

Contributor

I wonder why is this are the initialization messages printed in this method (it doesn't seem to have anything to do with the extensions).

@michalkurka

michalkurka May 25, 2017

Contributor

I wonder why is this are the initialization messages printed in this method (it doesn't seem to have anything to do with the extensions).

This comment has been minimized.

@mmalohlava

mmalohlava May 26, 2017

Member

Yeah, i can remove it but i did not want to break compatibility of interface.

@mmalohlava

mmalohlava May 26, 2017

Member

Yeah, i can remove it but i did not want to break compatibility of interface.

@michalkurka

LGTM; minor comments

mmalohlava and others added some commits Mar 4, 2017

[PUBDEV-4479] Modularization of H2O backend
This change contains two main code changes:
  - using SPI to register core extensions and REST API extension
  - removing use of reflection library for module lookup

  - We would like to provide more flexible system to extend H2O and plug new tools into the H2O platform (e.g, XGBoost, TensorFlow, Sparkling Water).
  - The current code base is using [reflections library](https://github.com/ronmamo/reflections) to handle lookup of optional components, however it brings several issues including:
    - limit on used package name by extension (only `water` and `hex` are allowed)
    - force traversal of full classpath which causes problems in systems with dynamic classloaders (e.g., Spark executors).

  - We will remove usage of reflections library to find instances of `water.AbstractH2OExtension`, `water.api.AbstractRegister` and `water.api.Schema`
  - The extensions (meaning classes listed in the previous point) will be registered using [Java Service Provider Interface](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html). In short, the concept relies on service files that are located in `META-INF/services` directory. Each service file is called by the name of a class it extends (e.g., `water.AbstractH2OExtension`) and contains a list of classes that extend the service class. For example, for core H2O REST API we have a single file `h2o-core/src/main/resources/META-INF/services/water.api.RestApiExtension` which contains 3 REST API extensions implementing interface `water.api.RestApiExtension`:
  ```
  water.api.RegisterResourceRoots
  water.api.RegisterV3Api
  water.api.RegisterV4Api
  ```

  - We provide capabilities REST end-point to provide list of registered core extensions, REST API extensions, parsers (WIP)

  - In the scope of H2O source code, we provide optional `@autoservice` annotation to register extensions (see [documentation](https://github.com/google/auto/tree/master/service)).

  - We do not modularize R/Python/Flow clients. The client is responsible to self-configure based on information provided by the backend and fails gracefully if the user invokes an operation that is not provided by backend

> Note: the same concept is already used in H2O to register parsers and Rapids extensions.

  - Code that register new REST API calls by extending `water.api.AbstractRegister` class will need to be updated by adding a service file as described above
  - Each class extending `water.api.Schema` needs to be registered as well in `water.api.Schema` service file.

----

History of changes
 - apt schema generation SPI files.
 - Proper schema registration
 - Service files
 - Fix service files
 - Schema registration as part of RestAPI registration.
 - AutoML modularized
 - Boot message provides a list of registered extensions for core and REST API.
 - Pick auto-service configuration from Kuba's branch.
 - Proper name for Rest API extensions.
 - Append missing schema into list of schemas.
 - More schemas.
 - Do not confuse test infrastructure by non-test files.
 - Integrating review comments

@mmalohlava mmalohlava merged commit fa3cd35 into master May 26, 2017

20 of 22 checks passed

Mr. Jenkins A1 Build finished.
Details
PR_pyunit_medium_large FAILED
Details
PR_R_cmd_check OK
Details
PR_R_cmd_check_as_cRan OK
Details
PR_junit OK
Details
PR_phantomjs OK
Details
PR_py3unit_medium_large OK
Details
PR_py3unit_small OK
Details
PR_pybooklets OK
Details
PR_python_demos OK
Details
PR_pyunit_small OK
Details
PR_rbooklets OK
Details
PR_rdemos_medium_large
Details
PR_rdemos_small OK
Details
PR_runit_medium_large OK
Details
PR_runit_small OK
Details
PR_runit_small_client_mode OK
Details
PR_smoke_junit OK
Details
PR_smoke_phantomjs OK
Details
PR_smoke_pyunit OK
Details
PR_smoke_runit OK
Details
PR_startup_checks OK
Details

@mmalohlava mmalohlava deleted the mm/exp/modularization branch May 26, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment