New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Root translation wrong after 2 seconds #42
Comments
Hi how "wrong" you are talking about the root translation? It is definitely not stable sometime. But it shouldn't be way off. |
Hi, It is quite bad to be honest. My sensation is that the scale by which I am multiplying the root translation for the first two seconds is correct, but then, it feels like it should be another one. |
Here is one for example |
I'm not be able to view this .fbx file in blender or any other online viewers. Something might be wrong with this file? Yeah the normalization is same across the entire sequence. So you don't need to treat other frames differently than the first two seconds. But the first two seconds are groundtruth motion, which serves as a seed to trigger the generation. So it is not superising that the following motions have worse quality Some of the results may have less satisfying results and some are better. Have you try other sequences? |
Hi, The .fbx file is ok. The thing is that it only contains the skeleton and the animation, not the model (otherwise it would be a much more hug We have tried several times most of motions generated by the evaluation test and they all have wrong root motion after two seconds. I really think that your method is awsome and the results are amazing, but for them to be practical and usable, there still this root motion problem to be solved. Maybe there is some other specific technique that can be dedicated to this? Otherwise, do you think that your method could be adapted to obtain better results for the root motion? Thank you very much, I really appreciate your help and your big work :) |
Hi for some reason I still can't view this fbx file, even in Autodesk. Could you sent me the raw output of the model, before converting to the fbx? It would save me some efforts on deploying this repo and running the inference. I'm currently a bit occupied so if you can share me the raw output I'm happy to take a look. Nevertheless, it is true the root translation is harder than the overall joint rotations to learn. |
Hi, As I said, there is no model in the fbx file. You'll find the character some units above the ground. Also, please find attached again the original output .npy file and the correspondent converted .fbx file. Thanks, |
I see haha. Thanks for the teaching!! Yeah the results are worse than I expected, not only the translation but also the joint rotations. I suspect our released model may not been fully trained comparing to the one we used for the paper. (This is a refactorized code base so the model is retrained after the submission). Also someone reports that retrain the model gives better performance. Here I attached some random results from our raw model. For your reference. Thanks for reporting this! I'm a bit full in the hands recently. I will look into this in the coming weeks. Best, |
Hi, I just saw your random results and they also have a lot of root problems to be practical animations. Let me know if I can help, Thanks, Alejandro Beacco |
Yeah those are very good ideas. There are also many cool ideas from this line of works: https://github.com/sebastianstarke/AI4Animation Also because of the auto-regressive inference, the error accumulate during inference. So another way to improve this is to train the model with auto-regressive scheme as well, a.k.a, using the inference motion to train the model. Unfortunately I don't have plans to push for this project in the near future. But feel free to try those cool ideas and post here with any progress! |
@abeacco Hi, how do you transfer .npy file to .fbx file? I've tried without success. |
We trained our model for ~ 3 days. The loss converged to the level of 1e-5 or 1e-6 (I can't clearly recall). It is important to train it long enough so that the model converged to a very good status. The inference is auto-regressive so error would accumulate which will be amplified if the model is not trained very well. |
I double-checked the tfrecord generation script in tools/preprocessing.py (LINE 167-182), for testval, you also random generate un-paired data:
which add 10 * 40 (20 val +20 test) unpaired-data to the val-test tfrecords. |
That's reasonable -- the results on the un-paired data is not as good as the results with paired motion seed and music |
seems 1e-4 ~ 1e-5 , not 1e-5 ~ 1e-6 |
Hello, I would like to ask how you got the translation scale of the root node? By Character Height? |
hi, have u solved the scale? |
This work is totally messed up, I advise you not to follow this paper. |
Hi,
Congratulations for such a great work.
We've been trying to recover the generated animations (in the output .npy) files to an fbx, and we are almost there.
Since you are normalizing the translation of the root, we are multiplying it again by the scale of our character in order to use.
However, this only seems to work for the first 2 seconds for every generated motion, after what the root translation gets kind of exagerated and wrong.
Any idea why this could be happening?
Thanks,
The text was updated successfully, but these errors were encountered: