Questions regarding State/Action inputs for GR00T and Pi0.5

[kc-shilin]

Hi!
First of all, thank you for your incredible work on FlashRT!

I recently tested it on my Jetson Thor and the performance is impressive!
I am planning to conduct further experiments and have a couple of questions regarding the roadmap and implementation:

1. ‎State input for Pi0.5
   After a quick review of the code and documentation, it seems only [Pi0/Pi0-FAST] currently supports state as an input,
   while the current API supports vision and prompts.
   Should I follow the guide of Adding a New Model to implement inference with state inputs,
   or do you have plans to officially add this feature in the near future?

2. ‎State and Action inputs for GR00T
   Similarly, I have a question regarding supporting state and action inputs for the GR00T model. Are there existing plans for this, or should we look into extending it on our end?

I'd appreciate any guidance or insights you can share. Thanks again for this awesome repository!
Thanks!

[LiangSu8899]

Thank you so much for testing it on Thor!

Yes, these inputs should eventually be supported. There are still quite a few features that are not fully aligned yet, such as GR00T N1.7 GPU support, state input support, FP8 / FP16 / BF16 path coverage, and some model-specific interfaces. They are all on the roadmap.

Right now, my main focus is to support more model types and architectures first, so that FlashRT can cover a broader range of kernels and execution patterns. As you can see from the current performance, the core value of the project comes from the reusable kernel/runtime library underneath.

Some smaller feature gaps and input interfaces are hard for me to cover quickly by myself, but they are definitely planned. The current priority is roughly:

1. support more models and expand the kernel library
2. improve the serving/runtime design
3. define more systematic inference templates and interface protocols
4. possibly support more hardware backends later

I will also try to fill in these model input/interface gaps as much as possible. Contributions are very welcome if you are interested in helping with any of these adaptations — feel free to open a PR or discussion anytime.

Thanks again for the feedback and for trying FlashRT on Thor!

[kc-shilin]

Thanks for the detailed clarification and for sharing the roadmap.

The current kernel/runtime performance is already very promising on Thor, so it’s exciting to see broader architecture support being prioritized first.

Also, it’s great to know that state/action inputs and the GR00T-related interfaces are already on the roadmap.

Thanks again for the great work and for being open to community contributions!

[LiangSu8899]

updated state handling to public VLA API in #50