Proposal: Unbundling Keras modules into separate repositories

[fchollet]

We should consider moving the modules preprocessing and applications to separate repositories under keras-team: keras-team/preprocessing and keras-team/applications.

They would be listed as a dependency of keras, and would still be importable from e.g. keras.preprocessing.

Why?

• They're not part of "the core Keras API" (which is about model building and training)
• They're not really coupled to the rest of the codebase, so unbundling is safe
• Faster CI runs, better test coverage
• Unbundling makes it easier to develop in an open-source setting
• Easier to sync across keras-team/keras and tf.keras (shared dependencies instead of code duplication)
• Other projects might have need of Numpy preprocessing utilities

Any comments or concerns? Who would be interested in taking greater ownership of these repos, should they materialize?

[fchollet]

We could also do the same with dataset. Less important though, since dataset is for debugging/demo purposes, and not for doing any real work.

[Dref360]

Minor concern :
Since preprocessing is where we push all of your data augmentation API,
should Data augmentation Layers (proposed https://github.com/keras-team/keras/pull/9056/files) be in the keras/preprocessing or since it's a layer, we should push it on Keras?

We should define clear boundaries for each repos. Also, will others repos be allowed to add dependencies?

And most examples in keras/examples will rely on all 3 repos. Should we add a separate setup.py for this folder to avoid circular dependencies?

[fchollet]

And most examples in keras/examples will rely on all 3 repos

They would just depend on Keras. Preprocessing/etc should still be importable from Keras. This change would be a pure refactoring, where the code is moved to a different place but the APIs don't change.

should Data augmentation Layers (proposed https://github.com/keras-team/keras/pull/9056/files) be in the keras/preprocessing or since it's a layer, we should push it on Keras?

For now I would suggest Keras contrib. But conceptually a layer would definitely be in Keras, not keras-processing, regardless of what it does. The question is thus whether we want data augmentation layers as part of the core Keras API.

[Dref360]

Okay, all fine then!

[srjoglekar246]

If anybody hasn't begun working on this, I would like to give it a go!

[fchollet]

The two new repositories will need "owners". An owner is someone who has write rights and bears responsibility for reviewing PRs (and merging if they think it should be merged), and to a large extent, for answering issues. I will still give guidance on API choices, but that will be limited to user-facing APIs decisions.

@srjoglekar246 which repo are you interested in working on, keras-processing, or keras-applications?

@taehoonlee would you be interested in becoming owner of the keras-applications repo, since you have done a lot of work on applications so far?

[fchollet]

@farizrahman4u @ozabluda @ahundt do you have any interest in getting involved too (see comment above)?

[srjoglekar246]

keras-processing sounds good then!

[taehoonlee]

@fchollet, I'd love to! I wonder if you intend to separate tests for the cores and the applications (as described in "Faster CI runs, better test coverage"). My minor concern is that the application tests will not be triggered whenever the cores are updated if the applications are unbundled.

[fchollet]

My minor concern is that the application tests will not be triggered whenever the cores are updated if the applications are unbundled.

Testing applications when changing layers would fall into "integration tests", not unit tests. Issues with the layers should be caught by the layers unit tests. We could add one application-related integration test to fix this.

[taehoonlee]

OK, I understand.

[srjoglekar246]

@fchollet Does it make sense to start work on this with a new personal repo and then move it all to one owned by keras-team?