Is torch rl going to support rl based Neural Architecture Search?

[jojo23333]

Hi, I'm wondering whether torchrl can be used as a simple framework for neural architecture search? The env is basically individual training &evaluation process on GPUs paralleled over potentially multiple nodes of GPU clusters.

[vmoens]

I guess we could write an example that shows how to do that.

What's the use case more specifically? What algorithm would you be using, reinforce? Ppo?

We can brainstorm about what the best API would be but TLDR we'd be thrilled to cover this!

[jojo23333]

For general use of NAS, I think reinforce ppo evolution are all good potential choices. Personally I was involved in an NAS search project with reinforce and I found there is no current framework for RL in NAS problems, so I feel this could fill the blank.

The core use case would be that:
1. I would sample archs from my controller(agent)
2. evaluate them by either Oracle training or other training free methods (This correspond to an ENV that handles training or other kinds of evaluation).
3. Use an collector to collect results from Evaluation threads, and then update my controller.

I think the only difference to the typical rl setup is that an Env may requires to be assigned to different GPUs (and potentially multiple gpu per Env). There is only one action required per env (which is the arch.) So generally I feel it makes sense to fit any NAS framework into torchrl :)

I'm happy to further discuss on this or contributing to the repo about the use case of RL for NAS :)

[vmoens]

This seems possible to cover.
Can I ask how you'd see the inference part? on a subprocess (like subprocess.run or similar)? Or a separate process using multiprocessing?
I guess that the "action" would be the hyperparameters represented as a (set of) discrete choices, the reward would be some metric measuring performance of the algorithm.
How many processes / workers would be needed? Is is something hugely distributed that you have in mind, or can it be handled by a single node (say with 4-8 gpus)?

I'd be happy to talk about the design of that thing!

[jojo23333]

I do write a framework by my own for my project on rl based nas. It spawns two separate process using torch.multiprocessing, one process handles the generation of action and training of the agent, another process handles the evaluation of archs (ENV process).

The ENV process would also be responsible for spawning and managing individual training process (number of training process is conditioned on number of available gpus, restricted to 1 gpu per training process so far). I'm not very familiar with how torchrl handles parallel envs, may be it is a similar logic?

In my case I'm still focusing on small dataset where I at most use 8 GPUs, so single node. But many practical neural arch (Efficient Net etc.) are searched on a larger scale where they report to use 64 GPUs so I assume it is on multiple nodes. My current framework only supports single gpu training NAS on single node. So I'm wondering whether sending jobs to multiple nodes are doable.

I can create a gist show some of my immature code framework for this if you feels its helpful XD