Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

About usage of text description #3

Open
EricGuo5513 opened this issue Jan 25, 2022 · 8 comments
Open

About usage of text description #3

EricGuo5513 opened this issue Jan 25, 2022 · 8 comments

Comments

@EricGuo5513
Copy link

Hi, thanks for your great work on ICCV. I have a question about the usage of text description. As we know, there could be multiple description for each motion animation. However, the proposed method is one-to-one mapping deterministic method. Do we also use mutiple descriptions for each motion during training, or we just keep using one specific description?

@anindita127
Copy link
Owner

Hi, thanks for your interest in our work. The method is trained with multiple text descriptions for a motion. You'll find the dataset has multiple annotations for the same kind of action so it's more of a many -to-one mapping. We also used a pose discriminator to train the model so it's not deterministic.

@EricGuo5513
Copy link
Author

EricGuo5513 commented Jan 25, 2022

Thanks for your reply. I have two further questions if you don't mind. During training, how do you deal with the variable length of training motions? During inference, does the proposed method use the ground truth initial pose, and motion length (Number of frames)?

@anindita127
Copy link
Owner

Hi, we downsample the training data to a fixed length. Yes, during inference the method uses the ground truth initial pose and we can mention the number of frames we want to generate (we have trained with 32 frames).

@Mathux
Copy link

Mathux commented Mar 4, 2022

Hi,

When you are evaluating your method against the ground truth motion, which text sequence do you use? (if there are several available for a particular sequence)

As your output has a fixed size, how do you compare with previous work for the evaluation?

Thanks for your help

@anindita127
Copy link
Owner

Hi, in the data preprocessing, we create a one-to-one mapping with the text description and pose sequence and use that as our ground truth data. Previous work like language2pose also has fixed output size. We calculate the mean position and variance metrics so evaluation can be done for a fixed sequence length for previous methods as well.

@Mathux
Copy link

Mathux commented Mar 4, 2022

Ok thanks, to be sure I get it, given a motion M1 with the annotations A1/B1/C1, M2 with the annotations A2/B2, you will create (M1/A1), (M1/B1), (M1/C1), (M2/A2) and (M2/B2)?

About the metric computation, I still did not understand. For example with the APE:

Let say hypothetically that:

  • the ground truth P^hat has 40 frames, so t varies between 0 -> 39.
  • your generation P^ghosh have 32 frames, so t varies between 0 -> 31.
  • language2pose P^jl2p have 35 frames, so t varies between 0 -> 34.

How do you compute "P^ghosh_t - P^hat_t"? What will we the "t"?

@anindita127
Copy link
Owner

Ok thanks, to be sure I get it, given a motion M1 with the annotations A1/B1/C1, M2 with the annotations A2/B2, you will create (M1/A1), (M1/B1), (M1/C1), (M2/A2) and (M2/B2)?

  • Yes, we create such pairs during data preprocessing.

How do you compute "P^ghosh_t - P^hat_t"? What will we the "t"?

  • During evaluation you can take fixed number of timesteps (for us t=32). We have downsampled the dataset and taken first 32 frames to train and evaluate. We have done the same with jl2p for comparison.

@Mathux
Copy link

Mathux commented Mar 4, 2022

Thanks it is much clear now.

I have another question about the number of data for the test set.
When I sample the sequences with the command:

python sample_wordConditioned.py -load save/ghosh/exp_124_cpk_2stream_h_model_nocurriculum_time_32_weights.p

I noticed that in the folder save/ghosh/exp_124_cpk_2stream_h_model_nocurriculum_time_32/test/ there is only 520 sequences. Whereas, in language2pose, I got 587 sequences for the test set. The 520 sequences are a subset of the 587.

Are you removing some sequences at some point?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants