-
Notifications
You must be signed in to change notification settings - Fork 260
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
This work does not outperform a Vanilla LSTM, given you sample one trajectory ? #8
Comments
For the second part, they have mentioned that a synthetic dataset was used in the original Social LSTM paper. This synthetic dataset was generated using a simulation. But the Social GAN paper does not use any synthetic data and trains on real data from scratch. |
It's unfair to say that this work gets outperformed by Vanilla LSTM (Ref Table 1 in the paper for the discussion below). |
@ashar-ali I have read the paper in some detail, I am well aware of this sentence which makes it a bit more unjustified. The author of Social LSTM[1] is a co author in this work and synthetic data is generated free of cost (automatically). ** The authors of Social GAN generated synthetic data themselves to demonstrate the effectiveness of pooling in Social GAN** (We would like to highlight that even though these scenarios were created synthetically, we used models trained on real world data.), but they did not use it to compare with social LSTM ? There is absolutely no point in not reproducing the results. I mean guys what is going on ? In this work Social LSTM is inferior to a trivial Vanilla LSTM. Just go through Social LSTM[1] and have a look at the margin it outperforms LSTM. @agrimgupta92 If you could kindly touch this aspect ? If results reported in Social LSTM are wrong, I am sure you had a discussion with Alahi regarding this aspect? @agrimgupta92 Not really, the original Vanilla LSTM (Alahi et al.) (baseline), proposed by your colleague basically sampled the next point through a Gaussian. Where as in the case of your work the Vanila LSTM is directly regressing the point, or at least this is the impression one gets out of reading your work. Now, you are giving more samples to your method to sample from and this advantage is not provided to Vanilla LSTM ? Could you perhaps comment ? I mean reading your response one gets the impression that one might stumble upon the parameters by luck but really using of adversarial training in this case is not really motivated, in my humble opinion. Please take it as a fruitful discussion from somebody who is student of the field. Looking forward to your in depth response especially regarding the results of Social LSTM. 1 Alahi, Alexandre, et al. "Social lstm: Human trajectory prediction in crowded spaces." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016. |
The quoted line is taken out of context (it is possible that on reading the paper it is unclear what we meant by synthetic in that context) . That line was specifically referring to the trajectories in Figure 5 of the paper. If you take a look at Figure 5 the scenarios don't require us to use human trajectory simulators based on Social Forces (it's trivial to make the input shown in Figure 5). The figure shows common "social" scenarios which our model learns. However, in contrast the Social-LSTM paper states they used a synthetic dataset for training which was generated using a simulation that implemented the social forces model. |
Its strange to say the least.
In early 2000s their was a tracker (single object), I do not exactly remember the name and they were doing something running experiment 100 times and taking the best and comparing it with the one ran single time.
Regarding the Social LSTM numbers, I understand you cannot comment, actually we were wondering that you perhaps could comment that what authors responded to you when you asked them, since they are/were at Stanford, in anyways I understand you cannot comment. [1]Ronny Hug et al. On the reliability of LSTM-MDL models for pedestrian trajectory prediction |
The focus of the paper is try to explain why it's beneficial to move away from just predicting ground truth trajectory. With that context I would like to clarify the evaluation concern you raised: |
The question remains unanswered. How are your numbers comparable to all the approaches you compare against ? Which other approach beside yours, access the ground truth information during inference time ? please address this question, I am asking it for the 3rd time. Regarding [28], you do not compare against it. No other trajectory forecasting method which you cite uses this evaluation protocol. Yours and your colleagues, own TrajNet challenge does not allow this kind of evaluation. Here is a quick test, take a training set, cluster 10 trajectories make a linear model (not even a regressor ), during test time, assign each individual the "best" trajectory. You will see how well Social GAN performs wrt a linear motion model. You will have your answer, if 20 is a small number or not. |
@agrimgupta92 Thanks for the efforts of trying social GAN, it's a great work anyway, at least it brought some interesting thoughts. However, I somehow agree with @AyeshaNirma. Although the problem is multi-modal, when we compare against other methods, we should only produce 1 sample and see if the accuracy is increased. Or we should a "multi-modal" evaluation tool, something like many acceptable ground truth trajectories, then the results make more sense (at least to me :)). |
@AyeshaNirma Thanks for pointing out the issues, I'm facing the same problems now. Did you find a better way to evaluate the model or a different solution to the problem of trajectory generation and evaluation? |
@agrimgupta92 The work of this paper is well done. @AyeshaNirma Good question! I also have the same confusion that why the author of this paper uses the best trajectory prediction that is nearest to the ground truth trajectory out of so many prediction results. This is just like when you are doing a multiple choice question which has only one correct answer, other models only choose one answer, but you predict a lot of potential answers which are called "socially-acceptable", and finally when judging right or wrong, you pick the one that is closest to the real answer in all your potential answers, and then claim that you make the right choice. This is definitely unfair to other models. I hope the author of this paper can tackle this problem in evaluation. |
This paper has some novelty. But publishing test time metrics conditioned on ground truth (which cannot be obtained in production) poses serious question on reproducibility and effectiveness in production. It also raises an ironic issue: If your model needs ground truth to perform well, why can't we just use the ground truth instead? Seems like the whole multi-modal trajectory prediction community adopts this practice since this paper. Would appreciate if any conference reviewer and committee could offer any insight on how they think about this. Glad that someone is willing to stand out with a comparision to classic time series const-velocity baseline: #44 (comment) |
Looking at the table 1 of your paper and under the column 1V-1, where you sample one trajectory and your numbers are actually potentially comparable to a Vanilla LSTM, since only in this case it is a fair comparison. This SOCIAL GAN work gets outperformed by a Vanilla LSTM ? Secondly, Alexandre Alahi the last author of your work and the first author of Social LSTM could not reproduce the Social LSTM results himself ? Could you please comment on these aspect ? No disrespect intended.
The text was updated successfully, but these errors were encountered: