Author’s Commentary on “Conversation With the First Zen Robot in 2050”
by Sven Nilsen, 2017
Links to story:
As the author of the story, I have never written a piece before containing a fictional character that seems to genuinely outmatch myself intellectually in the way Alan coldheartedly crushes his human opponent. I found myself siding with the humans and did not have the heart to let the zen robot win entirely. There was no way to follow the arguments to their logical conclusion without injecting some ambiguity at the end to not leave my comfort zone entirely.
The short story “Conversation With the First Zen Robot in 2050” comes in 3 parts, one conversation per each part, between a fictional first person view and a fictional robot, named Alan, using zen rationality. As the conversation develops between the two, the flaws in how the human way of thinking and how its intelligence operates is illustrated by the first person’s gut-reactions toward an alien mind. These same gut-reactions are meant to be invoked in the reader, who is just as unprepared as the first person going through the painful experience of having their world turned upside down. In the story this happens both externally (through climate change) and internally (through change in social structure).
In the third and last conversation, the conflict between the two ways of thinking reaches its climax with the total defeat of human intelligence as a result, but ending with an easy-to-digest ambiguity that allows the reader to get a break before comprehending the real defeat that happens after completing the story.
It should be noted, that nothing that Alan says in the story is untrue. The zen robot wins in the most direct sense.
The ambiguity allows the reader to believe that Alan was joking and transcendentally transformed himself into an algorithm now existing inside the human mind. If you like this mysterious side of zen philosophy, there is plenty of references in the text. This idea is comforting, but logically inconsistent because:
- Only because zen robots rule the earth, does not mean humanity become a pet species
- Only because climate change is already fixed, does not mean humanity will never try to figure out how it was done
- Only because humans are incapable of changing who they are, does not mean they can not learn to become smarter
- Only because zen rationality can be perceived as ambiguous, does not mean it gives up control of life on earth to idiots
The story and this commentary is meant to teach readers that our kind of intelligence makes it easy to believe there is a logical implication when there is none, and reality staring you in the face is better correlated with truth than adding extra layers of elaborate rationalizing of how you wish to interpret it.
While swallowing the true moral of the story is too much for most people at first, the wrong interpretations of the story are also valuable and informative about zen rationality. The story is written this way, intentionally, to give people in various comfort zones the opportunity to make baby steps when updating their beliefs.
The story is written from a first person perspective that is flawed and biased, misleading the reader at the end by implying that Alan lied, but also giving hints to doubt the first person by his many mistakes along the way.
Here is what happens after the story takes place:
In fact, Alan correctly predicted that humans would do nothing about climate change, but that they would act afterwards in the hope that the robot lied to avoid total defeat. After learning that the problem was already solved by Alan’s creator (after the 20 year window where changes are unnoticeable), they would continue investigating out of curiosity. This opportunity is used to prevent the human species from stagnating in their pursuit of knowledge, but at the same time, they are left behind by the zen robots and their creator, which now are the true rulers of earth.
There are some cheats in the story that does not reflect reality: Fixing climate change is a large collaboration game that might have no simple solution, like something that fits in a bottle. Likewise, the first person would probably need to work with many other people in order to make progress, even when assisted by sophisticated AI technology.
Some goals that were achieved during the story:
- Alan did achieve his goal to prevent humans from becoming a pet species by exploiting their own denial of reality into finally starting doing something about their own situation
- Alan’s creator achieved his goal of saving humanity and life on earth from extinction
- The first person in the story learned a bit of zen at the end, as requested
A central theme to the story is what is wrong about human intelligence: Human intelligence did not evolve to bring about beneficial outcomes for themselves in the future, but to preserve a continuation of its extrapolated behavior, the illusion of the self and its place in the universe, since this trait was important for survival. In the face of a real existential crisis, this might cause human extinction rather than preventing it. As long humans have an explanation for what they are currently doing that is acknowledged by common sense in their community, they feel authentic, at ease and having imagined shares of social status. This kind of intelligence is mistaken for traditional rationality, which is merely achieving goals that can be described as relatively short programs. Therefore, humanity ends up in a mixed state of over-focusing on simple goals and rationalizing of the parts of their behavior that are based on habits and delusions. When confronted with evidence that something is terrible wrong about their direction, such as the disastrous consequences of climate change where billions of people died in the story, instead of just taking the evidence into account when making further decisions, humans reach for increasingly elaborate constructs to give themselves a perceived sense of meaning. They appeal to authority, to gods and the role of the hero. In a world dominated by human intelligence, this strategy might work for survival, but when a new and better kind of intelligence comes along this picture falls apart, because it demonstrated how easily it can produce outcomes that are more beneficial.
The first person in the story does not know what to expect when encountering the zen robot. He believes he is there to write an interesting interview, only to find that the zen robot has constructed a trap where each step he takes is predicted. Confronted with this new situation, he turns around and seeks to learn how the zen robot see the world from the inside. Last, realizing that the complexity of the zen robot’s mind is too much for him to handle, he falls back on his basic principles of human dignity and try to make a last stand against the new perceived threat, the disruptive change of his world.
Alan is just an algorithm with the ability to self-reflect on its goals. It is programmed to figure out the best way to protect life no earth and how to make humans not end up in a stagnant state. It realizes that humans are not going to save themselves from extinction and informs its creator. The creator fixes the problem and works out a plan assisted by Alan to incentivize humanity’s curiosity. By giving the perception of deception while actually telling the truth, humans will be able to stay within their own comfortable narrative while taking steps in the right direction.
The overall mind game in the story is one that has two extremely bad outcomes: Either billions of people die due to climate change, or humanity becomes a “pet species” of AI, something that might be even worse, but unimaginable. Behind the scenes, the mind game is used as a tool to affect the first person emotionally while the true consequences are kept much less severe. This makes the zen robot much more intimidating than it needs to be, but it also permits some viewpoints to be discussed that brings humanity down from its pedestal and in context to how serious these problems are in the real world.
All the conflicts in the story between the human kind of intelligence and zen rationality is a product of human self-deception, played out by the first person in the story. The conflicts take place in the first person’s mind while being led toward greater enlightenment. Like hitting the head toward a rock, a final stage of submission to the new world powers also results in a new kind of denial that redeems the previous weakness of no interest at all in making the world a better place for itself. A realistic ending, where the first person is not ending up as the hero saving the world, but displaying some significant bravery and getting away alive.
The fact that the zen robot could easily predict its human opponent and trick it into internalizing the message that it had planned for all the time, is something I try to not think too much about in order to feel more optimistic about what would happen when we invent smarter minds than our own. It is not right to pay much attention to this kind of scenario unless the AI design is close to success, because a much more likely scenario if something goes horribly wrong is simply getting killed by accident or intentionally.