A reading list for a special topics class (R255) at the Computer Science and Technology Department at the University of Cambridge. This is part of Advanced Topics in Machine Learning.
The title of the course is:
Unconventional approaches in AI: complex systems perspectives, cognitive psychology, social sciences, computational models of creativity, explainable AI inspired by other disciplines and other unconventional models
This is AI or classical AI before big data. The time is now ripe to revisit these wonderful ideas and think about how to incorporate them in modern AI/deep learning. Insights from the past can inform future approaches to AI, especially in the age of big data.
Looking at the heritage of computing and its interdisciplinary past can inspire new approaches for the future. We need to learn lessons from the history of AI, what approaches worked and did not work in the past and how AI went through multiple winters.
These approaches can be used to develop techniques that can inspire explainable AI.
- Short introductory talk
https://www.youtube.com/watch?v=o7EXf265sTU
- Full talk
- Slides
https://github.com/neelsoumya/special_topics_unconventional_AI/blob/main/intro.pdf
https://github.com/neelsoumya/special_topics_unconventional_AI/blob/main/wrapup.pdf
- Computational models of creativity and scientific insight (Pat Langley)
https://escholarship.org/content/qt54x8v354/qt54x8v354.pdf
- BACON.5: the discovery of conservation laws (Pat Langley)
https://dl.acm.org/doi/abs/10.5555/1623156.1623181
- A Computational Inflection for Scientific Discovery
- Copycat (a computational model of analogy making)
https://dspace.mit.edu/handle/1721.1/5648
- The emergence of understanding in a computer model of concepts and analogy-making
https://www.sciencedirect.com/science/article/abs/pii/0167278990900865?via%3Dihub
- Structure mapping engine: Algorithms and examples
https://doi.org/10.1016/0004-3702(89)90077-5
- Learning new principles from precedents and exercises (Winston)
https://doi.org/10.1016/0004-3702(82)90004-2
- The overfitted brain: Dreams evolved to assist generalization
https://doi.org/10.1016/j.patter.2021.100244
- DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning
https://arxiv.org/abs/2006.08381
- On the measure of intelligence (Abstraction and Reasoning Corpus)
https://arxiv.org/abs/1911.01547
- Commonsense reasoning
https://dl.acm.org/doi/10.1145/2701413
Commonsense reasoning, Cyc and large language models
https://arxiv.org/pdf/2308.04445.pdf
Cyc database of commonsense reasoning (Doug Lenat and Gary Marcus)
http://web.archive.org/web/20230902080842/https://garymarcus.substack.com/p/doug-lenat-1950-2023
- Stories and narratives (Patrick Winston)
https://nautil.us/the-storytelling-computer-8380/
https://dspace.mit.edu/handle/1721.1/67693
- Argument technology for debating with humans
IBM Project Debater
https://www.nature.com/articles/d41586-021-00539-5
- Case based reasoning (Janet Kolodner)
http://alumni.media.mit.edu/~jorkin/generals/papers/Kolodner_case_based_reasoning.pdf
Can we (and should we) have consciousness in machines?
- Consciousness in Artificial Intelligence: Insights from the Science of Consciousness
https://arxiv.org/pdf/2308.08708.pdf
- Levels of AGI: Operationalizing Progress on the Path to AGI
https://arxiv.org/pdf/2311.02462.pdf
- 40 years of cognitive architectures: core cognitive abilities and practical applications
https://link.springer.com/article/10.1007/s10462-018-9646-y
Papers related to understanding and reasoning in large language models (LLMs): The philosophy of large language models
- Can large language models reason or understand?
https://arxiv.org/pdf/2210.13966.pdf
- Are they capable of consciousness?
- World models in large language models
https://arxiv.org/pdf/2310.02207.pdf
- Nature review
https://www.nature.com/articles/d41586-023-02361-7
- Sparks of Artificial General Intelligence: Early experiments with GPT-4
https://arxiv.org/abs/2303.12712
- A Theory for Emergence of Complex Skills in Language Models
https://arxiv.org/pdf/2307.15936.pdf
- LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations
https://arxiv.org/pdf/2305.18354.pdf
- Large Language Models as General Pattern Machines
https://arxiv.org/abs/2307.04721
- Hubert Dreyfus's crtique of Winograd's block worlds program
https://archive.org/details/whatcomputerscan017504mbp/page/n39/mode/2up
- Can LLMs understand? (Far from being “stochastic parrots,” the biggest large language models seem to learn enough skills to understand the words they’re processing.)
https://www.quantamagazine.org/new-theory-suggests-chatbots-can-understand-text-20240122/
- Large Linguistic Models: Analyzing theoretical linguistic abilities of LLMs
https://arxiv.org/abs/2305.00948
-
Theory of Mind benchmark for large language models
-
Are Emergent Abilities of Large Language Models a Mirage?
-
A Philosophical Introduction to Language Models
- Lessons for artificial intelligence from the study of natural stupidity
https://www.nature.com/articles/s42256-019-0038-z
- Explanation in artificial intelligence: Insights from the social sciences (Miller)
https://arxiv.org/abs/1706.07269
- Collective intelligence for deep learning: A survey of recent developments
https://journals.sagepub.com/doi/full/10.1177/26339137221114874
- Eric W. Bonabeau. Control mechanisms for distributed autonomous systems: insights from social insects. 2001
In Design Principles for the Immune System and Other Distributed Autonomous Systems.
https://www-users.cs.york.ac.uk/susan/books/pages/s/LeeASegel.htm#9582
(login with your RAVEN ID and search the university library webpage)
https://idiscover.lib.cam.ac.uk/
- Neural cellular automata
https://distill.pub/2020/growing-ca/
- Analogy in AI
https://melaniemitchell.me/PostdocProjectDescription.pdf
- Abstraction and Reasoning Corpus
https://github.com/fchollet/ARC
- Abstraction and Reasoning Corpus Challenge
https://blog.jovian.ai/finishing-2nd-in-kaggles-abstraction-and-reasoning-challenge-24e59c07b50a
https://github.com/alejandrodemiquel/ARC_Kaggle
Domain specific languages may be required (as suggested by Chollet) like genetic algorithms and cellular automata
- Other paths to intelligence elsewhere in the animal kingdom
https://nautil.us/another-path-to-intelligence-23113/
- The storytelling computer
https://nautil.us/the-storytelling-computer-237502/
http://web.archive.org/web/20221102094120/https://nautil.us/the-storytelling-computer-237502/
- The Psychology of Invention in the Mathematical Field (Jacques Hadamard)
https://archive.org/details/eassayonthepsych006281mbp/page/n35/mode/2up
- The Theory of Mind
https://en.wikipedia.org/wiki/Theory_of_mind
- The structure of scientific revolutions, Kuhn
https://github.com/Tijl/ANASIME
https://github.com/crazydonkey200/SMEPy
https://github.com/fargonauts/copycat
Present and lead a discussion on one of these papers (or any other related paper: come speak with me). The idea is that you raise some interesting questions. This course is meant to teach you research skills (like thinking critically about a paper and literature review skills).
In this course, each student would chose one paper. They would then do a presentation on it.
Towards the end of the term they would do a writeup/short report:
-
on this paper, and
-
the topic in general (unconventional AI). They would do a literature review of other papers in the field.
-
They will then reflect/write on how these techniques can be incorporated in modern AI/deep learning.
The intention is for students to learn how to read papers, and compare and contrast them to other papers and then evaluate what this means for AI/deep learning.
Some writing prompts for the writeup are here:
-
Short report (less than 4000 words). The idea is write a coherent narrative.
-
Suggest how these ideas can be incorporated in modern AI/deep learning systems
-
Why do you think these ideas were not successful in the 1950s/1960s?
-
What kind of data would we need to ensure these techniques would work today?
-
What lessons can we learn from the history of AI, what approaches worked and did not work in the past?
-
What could be the disadvantages of these approaches?
-
Rational reconstruction (analytical literature review/survey) of a research area
Other thoughts on the writeup:
-
A detailed research proposal with some ground work already accomplished
-
A hybrid of all of the above
Thoughts on a project:
- If you feel ambitious, you can do a coding project based on open source data (talk to me if you would like to do this).
https://www.cl.cam.ac.uk/teaching/2122/R255/
https://github.com/neelsoumya/special_topics_unconventional_AI/blob/main/admin_notes.md
- How to read papers
https://www.cs197.seas.harvard.edu/
- How to write
Write regularly
Keep a schedule
https://sites.google.com/site/neelsoumya/research-resources/scientific-writing
Video on writing
https://www.youtube.com/watch?v=DeVjXINr5Wk
Book on writing (please contact me to borrow a copy; also available from the library digitally)
How to Write a Lot: A Practical Guide to Productive Academic Writing by Paul J Silvia
You can also pick other papers that are broadly in this area/topic and that excite you. Please contact me to discuss further.
Soumya Banerjee
Office: FC01 (Computer Science and Technology Department)
https://sites.google.com/site/neelsoumya/Home
https://github.com/complexsystemslab/project_ideas/blob/main/project_ideas.md
