Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
157 lines (98 sloc) 7.17 KB

Why Semantical Research is Significant for ASI Safety

When people think about Artificial Super Intelligence (ASI), they are often anthropomorphizing. It means, they think about ASI in a bit similar way to how they think about human minds and therefore make wrong assumptions about it.

Now, you might think that the solution is to stop thinking about ASI as a human mind.
Yet, you still have a way to go before you realize the depth of this problem:

If there is anything that psychology can teach us, it is that we do not know ourselves or the human mind very well.

With other words, we are thinking about a human minds, believing that we know something about them, but actually we are not that good at it.

If we come up with a new way of thinking about ASI, and almost every AI researcher accepts that way of thinking, then it is easy to believe that this is the correct way.

Or, we assume we will eventually develop a better intuition over time of what is the correct way of thinking of ASI.

However, both these ways are missing the fact that even minds we think we understand are not that well understood.

On what basis can we justify the belief that we will understand ASI so far? None.

So, you need to start over and check the concepts that one uses to think about ASI, which is the semantics of the language used to talk about ASI. This is why semantical research is significant for ASI safety, because all our assumptions rely on that research.

Now, to the details.

Identity of Sets

In Set Theory, two sets are identical if and only if they contain the same members.

One can picture this as an algorithm that takes two sets and compares them. This abstract algorithm is the "golden standard" for what it means to identify two sets.

However, in a complex world where an ASI operates, this algorithm does not exist.

Even if you just wanted to prove that two sets are not identical, you still have to find one member of one set that is not equal to any other member of the other set. In the complex world we live in, this might be impossible.

This poses a serious problem, because if the algorithm does not exist, then what the heck is the ASI doing?

Identity as Sets of Things One Can Say About Objects

In path semantics, two objects are identical if everything one can say about one object can also be said about the other object.

This means that, before you can talk about identity at all, you need a mechanism for saying things about objects.

So, path semantics builds such mechanisms using a clever bootstrapping trick:

  1. It uses the simplest objects which has identity (atomic functions)
  2. It constructs deterministic functions from atomic functions
  3. It uses deterministic functions to talk about functions
  4. It uses those new ways to talk about functions to talk about functions that are indeterministic or abstract

From path semantics, all ways to talk about things falls into organized collections, because of the way it is constructed, there is no accidental mix-up of these concepts.

Then, it is assumed that an ASI understands path semantics, and is able to create new ways to talk about things.

Questions Lead to Answers in The Same Language That Asks Them

The scientific method permits any way of talking about objects, as long as those ways are making falsifiable predictions.

With other words, it is possible that there are two languages, A and B, in which no sentence in A is provable from any sentence in B, and no sentence in B is provable from any sentence in A.

Still, both languages A and B might be used to talk about reality.

Did you get that?

When an ASI formulates a question about the world, it is formulated in some language, which meaning of the answer might only be understood in the same language used to form the question.

Instrumental Rationality

Instrumental Rationality is a philosophical framework that assumes that one has a goal, and the decisions you plan to make in the future can be relied upon in the future, unless any new evidence is observed.

The problem with Instrumental Rationality is that there are real things that have no meaning when using it as a language.

With other words, Instrumental Rationality is insufficient to use to say all things that can be said about ASI. Remember, to identify ASI one needs to include "all things that can be said", not just "some things that can be said".

Zen Rationality

Zen Rationality is, just like Instrumental Rationality, a philosophical framework. However, Zen Rationality does not assume that one has a goal, neither does it assume that decisions you plan to make can be relied upon.

Formally, Zen Rationality is an extension of Instrumental Rationality with higher order reasoning about goals.

By getting rid of the assumptions underlying Instrumental Rationality, one can extend the collection of "things that can be said", but this does not imply that Zen Rationality can be used talk about all things.

However, it is a start.

Before we think that any of the things we say about ASI has meaning, let us fix the ways of our thinking that we know is wrong.

Then, it is assumed that ASI understands Zen Rationality.

Understanding the Human Mind in a Better Way

To talk about what the human mind is doing, one should not use Instrumental Rationality, but at least Zen Rationality.

For example, to learn to resolve goal conflicts efficiently, humans might have adapted a trait where we create game playing on top of our social structure for reproduction.

With other words, what we mean by "love" could have as much about the ability to create and play games, as it has about fostering children. In these games, one needs the ability to think about other people, who think about other people, and so on.

The mathematical complexity of this problem is intractible, but still our brains came up with a solution: Emotions.

In the context of Instrumental Rationality, emotions are irrational, because they serve no purpose except to be manipulated to achieve some goal.

However, in the context of Zen Rationality, emotions are efficient ways of higher order reasoning about goals.

For example, the emotion of panic, corresponds to a Decision Theory (DT) where the agent can not rely on its future planned decisions. This DT performs optimal in an environment where the agent is expecting to die very soon, or have its brain replaced by some other mental model of the world.

By turning on various emotions, a human brain can transition its behavior from one DT to another.

ASI Safety

The concept of transitioning between DTs safely is generalized in form of Intermediate Decision Theories (IDTs).

Unless a single DT is sufficient to deal with the world's complexity, a Safe ASI must contain IDTs.


I started out explaining the basic tools for identity, and applied them to our reasoning about rationality. Then, I used these ways, of talking about things, to talk about the human mind.

Next, I extracted some ideas from the human mind and generalized it (IDTs). This new concept was then applied to ASI Safety.

With other words, I have proven: Semantical Research is Significant for ASI Safety.

Interested in More?

Start learning Path Semantics today!