Questions for Jake Hoffman on his 5/16 talk on "An Illusion of Predictability in Scientific Results"

[jamesallenevans]

Pose your questions here for Jake Hoffman regarding his 5/16 talk on
An illusion of predictability in scientific results. In many fields there has been a long-standing emphasis on inference (obtaining unbiased estimates of individual effects) over prediction (forecasting future outcomes), perhaps because the latter can be quite difficult, especially when compared with the former. Here we show that this focus on inference over prediction can mislead readers into thinking that the results of scientific studies are more definitive than they actually are. Through a series of randomized experiments, we demonstrate that this confusion arises for one of the most basic ways of presenting statistical findings and affects even experts whose jobs involve producing and interpreting such results. In contrast, we show that communicating both inferential and predictive information side by side provides a simple and effective alternative, leading to calibrated interpretations of scientific results. We conclude with a more general discussion about integrative modeling, where prediction and inference are combined to complement, rather than compete with, each other. Contributing papers one and two.

[yiang-li]

How might this emphasis on integrating predictive information influence future research practices in fields that traditionally focus heavily on inference, like sociology where inference allows for testing the theoretical frameworks?

[XiaotongCui]

Thanks for sharing! You mentioned that even experts might confuse inference with prediction. How should we assist them in distinguishing between the two more effectively during the process of education and training?

[zcyou018]

Thanks for sharing! my question is how can the integration of explanatory and predictive models in computational social science improve the robustness and reliability of conclusions drawn from large-scale data analyses, and what are the potential pitfalls or limitations of such integrative modeling?

[zimoma0819]

Thanks for sharing! The study highlights a crucial gap in scientific communication by demonstrating that the exclusive focus on inferential uncertainty often leads to overestimations of treatment effects by experts. Incorporating both inferential uncertainty and outcome variability in data visualizations could help foster a more accurate understanding of scientific results among professionals. This approach could potentially revolutionize the interpretation of research findings, especially in fields where precise data interpretation is critical. My questions are whether this approach could become a standard practice and what we need to do to persuade people to adopt this approach to foster a good environment to present more accurate understanding of scientific results.

[bhavyapan]

Thanks for sharing your work! I am wondering about the implications of this awareness when it comes to the perception of research conclusions -- would experts be more likely to find certain observations or conclusions from research "believeable" if certain practices in communicating them are followed -- essentially could such approaches be possibly used as signalling tools in academic? How prevalent could this issue be across different scientific fields? In terms of policy, how could this illusion of predictability affect the manner in which academic research supports decision-making or feeds into legislative action?

[Dededon]

Thank you for the sharing! I’m curious in your insights about combining machine learning with causal inference research. How do you think of the theoretical frameworks as CausalBERT, that seems to sacrifice interpretability towards better estimation?

[Adrianne-Li]

Hello Dr. Hoffman,

Thank you for your insightful exploration of the illusion of predictability in scientific results. Your discussion raises important questions about the perception of research conclusions. Could the strategies you've proposed for communicating both inferential and predictive information serve as signaling tools within the academic community to enhance the believability of research findings? Additionally, how widespread do you believe this issue might be across different scientific disciplines? From a policy perspective, what impact might this illusion of predictability have on the way academic research is utilized in decision-making processes or influences legislative actions?

Looking forward to your thoughts,
Adrianne(zhuyin) Li
(CNetID: zhuyinl)

[kiddosso]

Thanks for sharing Dr. Hoffman! Your research is really inspirational. I wonder if you have pedagogical advice for teachers and professors in schools based on your paper An illusion of predictability in scientific results? Such a bias may be prevented in school and college education.

[oliang2000]

Hi Dr. Hoffman! In a class, we read a quote by John Rogers Searle that said, 'Prediction and explanation are exactly symmetrical. Explanations are, in effect, predictions about what has happened; predictions are explanations about what’s going to happen.' In your paper, you attempt to distinguish between these concepts. I'm wondering, would you say they are continuous/symmetric concepts or how do you think they relate to each other on a high level?

[C-y22]

Thank you for sharing! You discuss the potential for computational social science to advance by integrating predictive and explanatory modeling. However, one of the major challenges you highlight is the cultural and methodological differences between fields that traditionally focus on explanation and those that prioritize prediction (like computer science). Given this context, how do you envision overcoming these differences to foster collaboration between these disciplines? Specifically, what practical steps or policies could be implemented to encourage integrative modeling in research institutions or academic programs?

[66Alexa]

Thanks for sharing! What areas of future research do you believe are crucial to further understanding and addressing the illusion of predictability in scientific results?

[yuy123337]

Hi Dr. Hoffman! I am wondering why is prediction important in scientific research, particularly in the context of computational social science? Based on the discussed integration of predictive and explanatory modeling in computational social science, why is there an increasing emphasis on predictive modeling? What are the strengths of predictive approaches, and how do they complement traditional explanatory methods?

[Zihannah11]

Thank you for sharing. How do you think the emphasis on inference over prediction in various fields has influenced the way scientific studies are conducted, reported, and interpreted by both experts and the general public?

[natashacarpcast]

Hi! Thank you for this interesting research. I'm wondering which implications does this have for previous science? Would you say you're skeptical of everything that's already done? Or are there any areas where these problematics wouldn't be so harmful?

[iefis]

Thanks for sharing your research! Could you provide some more concrete examples that apply the integrative modelling approach in social science research?

[HamsterradYC]

Thanks for sharing your work! I'm wondering if the experiments described in the paper were conducted in controlled environments using hypothetical scenarios, can these scenarios accurately represent the complexities and challenges of everyday scientific communication?

[zhian21]

Thank you for sharing the interesting work. It explores how visualizing only inferential uncertainty in scientific findings can lead to misperceptions about the predictability and importance of these results. Through three randomized experiments, the study demonstrates that showing both inferential uncertainty and outcome variability leads to more accurate perceptions of scientific findings. The research highlights the need for better visualization practices to avoid overestimating the effects of scientific treatments and results. I am curious how the insights from this study can be practically implemented to improve the accuracy of scientific communication in academic publications and presentations.

[AnniiiinnA]

Hi Dr. Hoffman, thanks for sharing this interesting finding! In "Integrating Explanation and Prediction in Computational Social Science," you argue for the necessity of combining predictive and explanatory modeling to advance the field of computational social science. What specific barriers do you think contribute to the relative lack of integrative modeling research in computational social science? What strategies might be most effective in overcoming them to encourage more research that integrates predictive and explanatory approaches?

[Jessieliao2001]

Thanks for your kindly sharing! My question is : How does an emphasis on inference over prediction in scientific studies contribute to an illusion of predictability, and what are the implications for both experts and the general public in interpreting scientific results?

[JerryCG]

Dear Jake,

This is a very interesting statistically related pychological phenomenon. Indeed, as the sample size increases, the estimate, for example, the sample mean, will decrease greatly in the SD. The inferential uncertainty drops a lot. However, the SD for the outcomes should be relatively stable since they come from the same distribution. We are likely to be cheated by the seemingly significant difference in estimates and over-relies on this when predicting individual outcomes. The visualization does play a big role about how to form the correct understanding.

I wonder whether we can develop a new measure of the estimates in terms their predictability, adjusting the variation in the outcomes?

Best,
Jerry Cheng (chengguo)

[yunfeiavawang]

Thanks for sharing! The study reveals that focusing only on inferential uncertainty can cause experts to overestimate treatment effects. Could this approach become standard practice, and how can we persuade people to adopt it for better scientific communication?

[volt-1]

Thanks for sharing your insightful findings. In your research, you've shown how the way we present scientific data can affect how people understand the results, particularly when showing both the uncertainty of the estimates and the variability of individual outcomes. Could you discuss how using complex statistical methods like Bayesian hierarchical models, which show both these aspects, might change how people perceive scientific findings? Also, how might using these methods more regularly in reporting results impact public trust in science, particularly in fields with high variability like epidemiology or social science?

[shaangao]

Thank you for sharing your research! Given the results, what efforts can we make to support research groups with limited resources to better estimate individual treatment effects, on top of the traditional focus on average treatment effects?

[Kevin2330]

Hi Professor Hoffman,

I found your research fascinating, particularly regarding the balance between predictability and inference/interpretability, which is an area I'm also keenly interested in.

Question for 1st paper: How do you think visualizing both inferential uncertainty and outcome variability can be implemented in standard scientific reporting to minimize misperceptions among various audiences?

Question for paper 2: What are some practical challenges you foresee in integrating prediction and explanation within computational social science, and how can researchers overcome them?

Could you elaborate on the schema you proposed for thinking about research activities along the dimensions of causal effects and predictions? How can this schema guide future research directions?

Thank you!

[jiayan-li]

As computational social scientists, how can we reconcile the need for clear communication of inferential uncertainty and outcome variability, as highlighted in the first study, with the dual focus on prediction and explanation, as advocated in the second paper? Specifically, what strategies can we employ to ensure that our visualizations and models not only accurately convey the uncertainty and variability in our data but also effectively integrate causal explanations and predictive power to enhance the overall interpretability and utility of our findings?

[yunshu3112]

Hi Dr. Hoffman, I am very interested in your findings in your paper "An illusion of predictability in scientific results". I wonder how you interpret the tension between causality and predictability. In addition, do you think future improvement of statistical tools can mitigate this illusion of predictability by enhancing inferential certainty?

[adamvvu]

Thanks for sharing your work. It highlights the importance of interpreting the point estimates and uncertainty in the context of the outcome's variability. I wonder however if the results from the experiments were more of a cognitive bias related to scale? Were all three experiments on the same scale?

[Weiranz926]

Thanks for your sharing!
Given the emphasis on presenting both inferential and predictive information to provide a more calibrated interpretation of scientific results, what practical steps or guidelines would you recommend for researchers to ensure their studies effectively balance these two aspects, especially in fields where prediction has traditionally been undervalued?

[Yuxin-Ji]

Thank you for sharing! It's very interesting to learn about this psychological phenomenon on how people perceive the effect when presented with different visualizations. Do you think there are any con side for presenting the inference and prediction together, such while they avoid overestimating the effect, people are less confident in reaching a conclusive finding?

[yuzhouw313]

Hello Professor Hoffman,
Thank you for sharing your work with us. While I understand that inferential uncertainty and outcome variability focuses on inference and prediction respectively, where the former draws conclusion from sample to population and the latter predict future events given current/past events, are there special cases of the overlapping of these two? Specifically, could you elaborate on how both inferential uncertainty and outcome variability impact the reliability and validity of conclusions in longitudinal experiments where both internal and external factors play a significant role over time?

[franciszz992]

Your papers are very inspiring and interesting to read. Can't wait to see the presentation tomorrow. I'm convinced of the suggestions that you set out in the Perspective paper, and I'm particularly interested to see your argument to use a common task framework to centralize the collective efforts in a research field. Has it ever been sucessfully implemented? Who should be centralizing the research? And does it make (political) economic sense for such a centralized framework/institution to exist?

[yuanninghuang]

Could you elaborate more on the specific methods or formats that you have found most effective in achieving the balance of presenting both inferential and predictive information side by side for more calibrated interpretation of scientific results, essentially in communicating with experts and non-experts in the field?

[ksheng-UChicago]

Thanks for sharing. The research result is very interesting. I wonder if such research can be helpful in both academia and industry. Definitely would like to learn more about it.

[Joycepeiting]

Thank you for sharing! Paper 2 mentions how structural models in economics are positioned in quadrant 4, Integrate modeling, as they are capable of conducting predictive counterfactual analysis. However, many of these models rely on specific theoretical and statistical assumptions. What are your thoughts on the tradeoff between the flexibility of these underlying assumptions, which may impact generalizability, and the predictability built on causal statements?

[YutaoHeOVO]

Hi Mr Hoffman,

I think the key of your argument is on inference versus prediction. The emphasis of inference actually originates from the requirement of yielding causal interpretations instead of simply predicting the dynamics. (Researchers tend to care more about what is behind the data instead of what can be directly observed from the data.) Indeed, it can be the case that some causal findings might be trivial, yet I am afraid your concern on predictability and misperception in scientific finding is simply emphasizing the trivial facts that many social scientists, or at least economists are already well acquainted of.

[lguo7]

Thank you for sharing your research! In the discussion, you suggest that displaying both inferential uncertainty and outcome variability can reduce the “illusion of predictability.” Considering the practical constraints of visualizing large datasets or highly skewed data, what alternative visual representation strategies would you recommend to effectively communicate these statistical concepts?

[hchen0628]

Thank you for the sharing! In light of the findings on the illusion of predictability in scientific visualizations, how can researchers effectively communicate the limitations of their data without undermining the perceived validity and impact of their scientific contributions?

[MaxwelllzZ]

Thank you for sharing. Dr. Hoffman, in your researches, you've emphasized the importance of integrating explanatory (inferential) and predictive methodologies in computational social science to enhance the clarity and applicability of research findings. Could you elaborate on how this integrative modeling approach has influenced the methodologies and outcomes of your recent projects? Additionally, what challenges and opportunities do you see in encouraging broader adoption of this approach within the field?

[Caojie2001]

Thank you for sharing your interesting research! I wonder are there any example projects that you consider successful in integrative modeling, effectively combining prediction and explanation?

[Brian-W00]

The first paper revealed that displaying inferential uncertainty alone may lead even highly trained experts to overestimate treatment effects. How does displaying a combined visualization of inferential uncertainty and predictability of outcomes change professionals' understanding of scientific results?

[xiaowei-v]

It is a really interesting topic. The first paper shows people should be able to interpret the results more accurately with both inferential uncertainty and outcome uncertainty. And the second paper stress that we can provide people with more explainable results. This reminds me of some research on AI aversion. I would expect people demonstrate greater acceptance if they have access to explanations that help they understand the decision process. However I am curious how should we make some of the process say prediction explainable when it may require domain knowledge to fully grasp the mechanism.

[nalinbhatt]

In agent-based modeling there is ODD+D protocol to convey results and portray model complexity. Could having a set of agreed upon protocols to communicate results mitigate some of these issues.

[DonnieTang1]

Thank you for sharing. How does focusing on inferential uncertainty over outcome variability create misperceptions about the implications of scientific results

[Marugannwg]

I'm intrigued with the idea of further getting down the line of what "predict" means when many times we confuse it with finding significant evidence to support/explain a phenomenon under a (single) schema. From a researcher's perspective, what are some common goals that are overlooked? How do we tinker our research process toward particular goals? And what is this ideal landscape of research space when most studies follow the open-science guideline you posed in the paper?

[Ry-Wu]

Hi Dr. Hoffman, thank you so much for sharing your interesting work! I'm wondering if there are some specific challenges and potential biases that arise when scientific findings are presented primarily through inferential statistics?

[kunkunz111]

Thanks for your great work! I am curious about the implications of your findings for the broader scientific community. Specifically, could adopting a more nuanced approach to distinguishing correlation from causation enhance the reliability of scientific conclusions? Additionally, what challenges might researchers face in implementing such approaches, and how can we overcome these barriers to improve the accuracy and trustworthiness of research findings across various scientific disciplines?

[zihua-uc]

Thank you for sharing your work! I wonder whether there is significant heterogeneity of this effect across fields (e.g., economics, psychology, sociology)?

[WonjeYun]

Thank you for presenting your research. There has been research in various fields trying to explain the outcome of ML models. Especially, social scientists use causal inference as their main method. Do you think whether there needs improvement?

[bairr1208]

Thank you for sharing your inspiring research, Dr. Hoffman. Your study on the illusion of predictability in scientific results is fascinating and raises important questions. How do you think the awareness of this issue affects the perception of research conclusions among experts? Specifically, could certain communication practices serve as signaling tools to make observations or conclusions appear more believable across different scientific fields? In terms of policy, how might this illusion of predictability influence the way academic research supports decision-making and legislative action? Additionally, do you have any pedagogical advice for teachers and professors to help prevent such biases in school and college education? Lastly, how can the insights from your study be practically implemented to improve the accuracy of scientific communication in academic publications and presentations?

[vigiwang]

Thanks for sharing! How should we trade off causality and predictability in our research? Is there a reliable way for ML to guide us to understand the underlying mechanisms behind social behaviors? Or should we always use it to identify important features?

[binyu0419]

Thank you for your presentation, you highlighted the long-standing emphasis on inference over prediction in scientific studies and how this can mislead readers about the definitiveness of results. Could you elaborate on the specific methods or approaches that can effectively integrate both inferential and predictive information in scientific reporting?

[Daniela-miaut]

Thank you for your research! You mentioned in your paper that, by recognizing the limitation of predictive accuracy in social science, the presence of system complexity or intrinsic randomness can be yield evident. How do you think can we possibly integrate complexity with integrative models?

[xinyi030]

Thanks for sharing Dr. Hoffman! Considering the challenges in balancing prediction and inference, what practical steps can researchers take to ensure that their findings are communicated effectively, minimizing misinterpretation without oversimplifying complex data?

[lbitsiko]

Fascinating insights about the significance of inscriptions, immutable mobiles, visualizations in a sort of Latourian way. What would you think is the role of math education in misinterpretation?

[icarlous]

Thank you for your insights! You mention how computational social science could progress by combining predictive and explanatory modeling, but a key challenge is the cultural and methodological divide between fields focused on explanation and those prioritizing prediction, like computer science. How can we bridge these gaps to foster interdisciplinary collaboration? What practical steps or policies could research institutions or academic programs implement to promote integrative modeling?

[Aiwen-Xiao]

Thank you for sharing your insightful research. Given your findings on the misinterpretation of inferential uncertainty as outcome variability, how can researchers and practitioners better distinguish and communicate these concepts in their visualizations?

[cty20010831]

Thanks for sharing! I am wondering what practical steps researchers can take to avoid creating an illusion of predictability in their studies. How can educators and institutions incorporate your findings into scientific training and curriculum?

[Huiyu1999]

Thanks for sharing! My questions are: What practical steps can journals and researchers take to incorporate both inferential uncertainty and outcome variability in their visualizations to improve the clarity and accuracy of scientific communication? How can future studies build on the findings of this paper to further investigate the impact of visualizing both inferential uncertainty and outcome variability on various audiences?

[Yunrui11]

Thank you for sharing your insights through these abstracts. I found the concept of integrative modeling particularly intriguing as it suggests a balanced approach to incorporating both predictive and explanatory aspects of research. Given your extensive background in both academia and industry, how do you see the integration of prediction and explanation evolving in the field of computational social science, particularly with the ever-increasing availability of big data? Additionally, what are some of the practical challenges you foresee in achieving effective integrative modeling, and how might we, as emerging scholars, prepare to tackle these challenges?

[aliceluo1]

Thank you so much for sharing! Given the challenges associated with balancing inference and prediction in scientific studies, what practical steps can researchers take to ensure that their findings are communicated in a way that appropriately reflects both aspects, thereby reducing the illusion of predictability for both experts and lay audiences?