Harold/ch773/decompose training mechanisms into a suite #106
Conversation
The notion of an `epoch-processor` is a good one. The problem is that the responsibilities around reporting and processing are spread around the entire training process. This is the first step (of likely many) to simplify high-level training by empowering the epoch processor to extensibly handle training regimes. Training then can be pretty dumb, and a pluggable set of epoch processors can handle the demands of various training scenarios.
For consistency and greppability.
|
@cnuernber --- Just starting on the training reorg. This commit is the interesting one: cc: @poojaramesh @rosejn @charlesg3 --- also potentially relevant to your interests. |
|
One thing that made that train-n much harder was trying to provide a resource context wrapping everything. If we remove that constraint and just provide good documentation around using it then I think it isn't necessary to do such an inversion of control. During the develop of this, however, because I saw every engineer making the exact same mistake of not using a resource context and because I got so many questions about them it was clear that the understanding wasn't there to expose it. Without that understanding I think an inversion of control is unavoidable. With that understanding I think it is completely unnecessary to do more than provide some set of utility functions and let people do various combinations of map, take, and take-while that would in effect to everything we do now. So enforcing that people know and respect the resource context issue means we can open up a lot more flexibility. Without that we have to invert control eventually which probably leads back to the same point. |
|
I think that as long as the substrate that this particular layer of the
cake is built off of suitable building blocks (push some things down) and
some of the higher level functions (such as saving the network) are pushed
up into utility functions that can be called, then we can certainly support
both -- the inversion of control where you pass some additional pipeline
elements / middleware for saving, deciding when to exit and what not is a
pretty cool abstraction for building up more advanced training procedures
-- but as long as that is built on something sensible, the end user also
has the ability to respect the constraints around the resource context and
build their own training loop more explicitly... possibly using this
interface as an example. Really we just need to nail the contract with the
lower level to make it clean and easy to consume and what we build on top
of that can be pretty flexible.
/$0.02
…On Thu, Feb 23, 2017 at 3:13 AM, Chris Nuernberger ***@***.*** > wrote:
One thing that made that function harder was trying to provide a resource
context wrapping everything.
If we remove that constraint and just provide good documentation around
using it then I think it isn't necessary to do such an inversion of control.
During the develop of this, however, because I saw every engineer making
the exact same mistake of *not* using a resource context and because I
got so many questions about them it was clear that the understanding wasn't
there to expose it. Without that understanding I think an inversion of
control is unavoidable.
With that understanding I think it is completely unnecessary to do more
than provide some set of utility functions and let people do various
combinations of map, take, and take-while that would in effect to
everything we do now.
So enforcing that people know and respect the resource context issue means
we can open up a *lot* more flexibility. Without that we have to invert
control eventually which leads back to the same point.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#106 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFzjVdGLReal3vKTXn0ReNQl62qgG_plks5rfVu4gaJpZM4MJVKa>
.
--
*Charles Gruenwald, PhD*
*Director of Engineering /Founder*
1050 Walnut St. Suite 500
Boulder, CO, USA, 80302
office: (720) 432-4428 <+(720)+432-4428>
thinktopic.com
|
|
@cnuernber @charlesg3 good thoughts. I agree that the context problem is intertwined here, but I think it's soluble separately (there's a separate story for it on clubhouse). I think the way forward here is incremental improvements, which I'll continue prodding. Perhaps when you're back next week, Chris, we can agree on some more sweeping changes (I'd like to eliminate the |
|
Is this PR ready for a merge back? What is going on here? |
|
Yeah, I think merging this is a good idea. There's still a lot of work to do, but this is on the path imo. |
cnuernber
deleted the
harold/ch773/decompose-training-mechanisms-into-a-suite
branch
* [suite/train/train-n] Rewrite this docstring. * [suite/train] Begin to separate concerns around `train-n`. The notion of an `epoch-processor` is a good one. The problem is that the responsibilities around reporting and processing are spread around the entire training process. This is the first step (of likely many) to simplify high-level training by empowering the epoch processor to extensibly handle training regimes. Training then can be pretty dumb, and a pluggable set of epoch processors can handle the demands of various training scenarios. * [suite/classification] Rename: best-network-function -> best-network-fn For consistency and greppability. * [suite/train] Removing unused function. * [suite/train] A lying docstring and some trivial cleanups.
No description provided.