You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The design of MutationOps (contents of op.jl) are a result of now obsolete design choices. In the current design (post #37) they are confusing and add bloat.
Not sure exactly how to clean them up as some kinds of "future size" metadata is still needed for output selection, mostly due to #40 which I think prevents implementing something like "select or insert outputs based on this new desired size".
Currently the library also tries to cater for the case when one does not want to prune an existing model but instead just change the sizes of some architecture "template" or "specification" which generates new models from scratch.
The current design kinda silently handles both without making any assumptions on which one the user wants to do. However, applying a size only mutation to an actual network might cause severe performance degradations as outputs are then misaligned with inputs.
If some "select or insert outputs based on this new desired size" can be created I hope this would allow for just letting the same API call (ie deltaN{in,out}) perform size change or outputs selection based on what the vertex represents (e.g. an actual layer with existing weights or an architecture spec).
The text was updated successfully, but these errors were encountered:
Remove the temporary/intermediate size concept altogether so that any size/structure change always modifies the parameters (or at least makes the call to the underlying implementation). Apart from the overall yuckiness and inconvenience of doing size-change and then have to remember to call outselect it does not work in all cases, Keeping track of and trying to educate users (i.e make me remember) when it works and not is an unecessary burden.
To facilitate the "don't discard parameters if you intend to change several vertices" I'll try to make an API where one can alter the size of several vertices in one go and let the solver figure it out.
For the architecture spec vs actual model, I will try to ask the underlying implementation and default to full selection.
The design of MutationOps (contents of op.jl) are a result of now obsolete design choices. In the current design (post #37) they are confusing and add bloat.
Not sure exactly how to clean them up as some kinds of "future size" metadata is still needed for output selection, mostly due to #40 which I think prevents implementing something like "select or insert outputs based on this new desired size".
Currently the library also tries to cater for the case when one does not want to prune an existing model but instead just change the sizes of some architecture "template" or "specification" which generates new models from scratch.
The current design kinda silently handles both without making any assumptions on which one the user wants to do. However, applying a size only mutation to an actual network might cause severe performance degradations as outputs are then misaligned with inputs.
If some "select or insert outputs based on this new desired size" can be created I hope this would allow for just letting the same API call (ie deltaN{in,out}) perform size change or outputs selection based on what the vertex represents (e.g. an actual layer with existing weights or an architecture spec).
The text was updated successfully, but these errors were encountered: