0.6 Rarity

Working concept — not standard information-theory terminology

Rarity is a pedagogical model used in Data to make the transition from probability to Shannon information more visible.

The formal theory does not require this quantity. Its purpose is to expose the structure hidden inside:

$$ I(x)=-\log_2 p(x) $$

---

🏵 The idea

Probability tells us how expected an observation is.

For an observation x:

$$ p(x) $$

large probability means the observation was easy to expect.

Small probability means the observation had to cross a much narrower path through possibility.

Instead of immediately applying a logarithm, consider first taking the reciprocal:

$$ R(x)=\frac{1}{p(x)} $$

We will call this quantity Information Rarity.

It is dimensionless.

Its purpose is to provide a mental model:

The less probable an observation is, the more it resists our prior expectation.

---

🔰 Probability and Rarity

Probability lies between 0 and 1:

$$ 0 < p(x) \leq 1 $$

Information Rarity therefore lies between:

$$ 1 \leq R(x) < \infty $$

The mapping is simple:

Probability       Rarity

1                 1
1/2               2
1/4               4
1/8               8
1/16              16
1/1024            1024

So:

$$ p(x)\downarrow \quad\Longrightarrow\quad R(x)\uparrow $$

A highly probable observation has little Rarity.

A rare observation has large Rarity.

Rarity: Operational Interpretation

If an event occurs independently on every trial with probability (p), the expected waiting time until it occurs is:

$$ \boxed{ E[T]=\frac1p=\rho } $$

A rarity of 10 means that, under repeated independent opportunities, you wait 10 trials on average for the event.

Rarity has three mutually reinforcing interpretations:

• $R(A) = 1 / P(A)$
  • Rarity: "one in ρ"
  • Recurrence: About ρ trials between occurrences
  • Resistance: reciprocal of probabilistic conductance

Probability has no dimension or unit, but it has a meaning as recurrence of cases, therefor Rarity is interpreted as a separation or cycle length of this recurrence, which in average is true.

🏵 Certainty has minimum Rarity

Suppose:

[ p(x)=1 ]

Then:

[ R(x)=\frac{1}{1}=1 ]

This is the minimum possible Rarity.

Nothing had to be overcome.

The model already predicted the observation with certainty.

Later:

[ I(x)=\log_2R(x) ]

therefore gives:

[ I(x)=\log_2 1=0 ]

So a Rarity of 1 corresponds to zero information gained.

---

🔰 Rare observations create large Rarity

Suppose instead:

[ p(x)=\frac{1}{8} ]

Then:

[ R(x)=8 ]

And Shannon information becomes:

[ I(x)=\log_2 8=3\text{ bits} ]

The transformation can therefore be seen in two stages:

p(x)
 │
 │ reciprocal
 ▼
R(x) = 1/p(x)
 │
 │ logarithmic scale
 ▼
I(x) = log₂ R(x)

Or:

[ \boxed{ I(x)=\log_2R(x) } ]

with:

[ \boxed{ R(x)=\frac1{p(x)} } ]

---

🏵 Why introduce Rarity at all?

The usual expression:

[ I(x)=-\log_2p(x) ]

can hide two different transformations.

First:

[ p(x) \rightarrow \frac1{p(x)} ]

turns rarity into magnitude.

Then:

[ \frac1{p(x)} \rightarrow \log_2\frac1{p(x)} ]

turns that magnitude into an additive scale.

So Shannon information can be understood as:

the logarithm of the Rarity of an observation to expectation.

This makes the negative sign disappear conceptually:

[ -\log_2p(x)

\log_2\frac1{p(x)}

\log_2R(x) ]

---

⚙️ Rarity is multiplicative

The most important reason this intermediate representation is useful appears when independent observations are combined.

Suppose:

[ P(x,y)=P(x)P(y) ]

Then:

[ R(x,y)

\frac{1}{P(x,y)} ]

and therefore:

[ R(x,y)

\frac1{P(x)P(y)} ]

so:

[ \boxed{ R(x,y)=R(x)R(y) } ]

Independent observations multiply Rarity.

---

🏵 Shannon information linearizes Rarity

Now apply the logarithm:

[ I(x,y)

\log_2 R(x,y) ]

Since:

[ R(x,y)=R(x)R(y) ]

we obtain:

[ I(x,y)

\log_2[R(x)R(y)] ]

and therefore:

[ \boxed{ I(x,y)=I(x)+I(y) } ]

This is one of the most important structural transitions in Information Theory:

Probability
    multiplies

        ↓ reciprocal

Rarity
    multiplies

        ↓ logarithm

Information
    adds

Or more compactly:

[ p_1p_2 \quad\rightarrow\quad R_1R_2 \quad\rightarrow\quad I_1+I_2 ]

⚜️ The logarithm converts a multiplicative structure into an additive one.

---

💡 Binary Rarity

The connection becomes especially visual when probabilities are powers of two.

Suppose:

[ p(x)=2^{-n} ]

Then:

[ R(x)=2^n ]

and therefore:

[ I(x)

\log_2 2^n

n ]

So:

[ \boxed{ p=2^{-n} \Rightarrow R=2^n \Rightarrow I=n\text{ bits} } ]

Examples:

p = 1/2       R = 2       I = 1 bit

p = 1/4       R = 4       I = 2 bits

p = 1/8       R = 8       I = 3 bits

p = 1/16      R = 16      I = 4 bits

p = 1/1024    R = 1024    I = 10 bits

Every additional bit corresponds to doubling the Rarity.

Equivalently:

One bit distinguishes between possibilities whose probability differs by a factor of two.

---

🪄 Information as orders of Rarity

This interpretation gives another useful intuition.

Information is not proportional directly to rarity.

It measures the order of magnitude of rarity on a chosen logarithmic scale.

For base 2:

[ I(x)=\log_2R(x) ]

asks:

How many doublings of Rarity separate this observation from certainty?

For example:

R = 1       → 0 doublings → 0 bits
R = 2       → 1 doubling  → 1 bit
R = 4       → 2 doublings → 2 bits
R = 8       → 3 doublings → 3 bits
R = 1024    → 10 doublings → 10 bits

This is perhaps the strongest visual interpretation of the bit.

---

☯ Probability and Rarity are dual views

Probability and Information Rarity contain exactly the same numerical information:

[ R=\frac1p ]

and:

[ p=\frac1R ]

Nothing new has been mathematically created.

The value of R is representational.

Probability answers:

How available was this outcome?

Rarity answers:

How strongly did the distribution suppress this outcome?

They are reciprocal views of the same model.

---

🏵 From possibility to information

This gives a useful conceptual sequence:

Possibility
    ↓
Probability
p(x)

How strongly was this outcome expected?

    ↓

Rarity
R(x) = 1/p(x)

How far from expectation was it?

    ↓

Information
I(x) = log₂ R(x)

How many binary orders of surprise does that Rarity represent?

The intermediate step makes Shannon's formula less mysterious.

---

⚠️ Do not confuse this with electrical Rarity

The analogy with electrical circuits is useful, but limited.

Electrical Rarity obeys physical composition laws.

For example, resistors in series satisfy:

[ R_{\text{series}}

R_1+R_2 ]

Information Rarity does not behave that way.

For independent observations:

[ R(x,y)=R(x)R(y) ]

not:

[ R(x,y)=R(x)+R(y) ]

So Information Rarity should not be treated as an electrical Rarity measured in ohms.

It is a dimensionless reciprocal probability.

The analogy concerns the intuition of opposition or difficulty, not the complete mathematics of electrical networks.

---

⚠️ No parallel-resistor interpretation yet

It may be tempting to search immediately for analogues of:

• resistors in series
• resistors in parallel
• current
• voltage
• conductance
• impedance

This should be resisted unless a mathematically exact correspondence is found.

A metaphor becomes dangerous when its decorative structure begins generating false mathematics.

For now, preserve only the useful correspondence:

High probability
≈ low Rarity

Low probability
≈ high Rarity

Independent probabilities multiply
≈ Raritys multiply

Logarithm of Rarity
≈ additive information

Nothing beyond this is currently required.

---

💡 Conductance as probability?

If the analogy is explored further, there is an interesting inversion.

Electrical conductance is:

[ G=\frac1R ]

Under the proposed definition:

[ R(x)=\frac1{p(x)} ]

therefore:

[ G(x)=p(x) ]

So probability itself occupies the mathematical role analogous to conductance.

This produces an attractive conceptual pair:

Probability        ↔ conductance
Information
Rarity         ↔ Resistance

or:

[ p=\frac1R ]

⚠️ Again, this is a structural analogy only.

It should not be interpreted as an electrical model of probability.

---

🔥 A possible deeper interpretation

The resistance metaphor suggests a useful statement:

An observation is informative when it crosses a region of possibility that the model assigned little conductance.

Or more simply:

Expectation offers little resistance to the ordinary and great resistance to the rare.

Shannon information then does something subtle.

It does not report Rarity directly.

It reports its logarithm:

[ I=\log_2R ]

This compresses enormous differences in probability into an additive scale that can accumulate across independent observations.

That may be the key pedagogical purpose of this concept.

---

🏵 Relation to entropy

Information Rarity belongs to individual possible observations:

[ R(x)=\frac1{p(x)} ]

Surprisal is:

[ I(x)=\log_2R(x) ]

Entropy then remains the expected surprisal:

[ H(X)

\mathbb E[I(X)] ]

Therefore:

[ H(X)

\mathbb E[\log_2R(X)] ]

or, in the discrete case:

[ H(X)

\sum_x p(x)\log_2R(x) ]

Since:

[ R(x)=\frac1{p(x)} ]

this is equivalent to Shannon entropy:

[ H(X)

-\sum_x p(x)\log_2p(x) ]

So entropy can also be read as:

the expected logarithmic Rarity encountered when an observation is resolved.

This interpretation should remain secondary to the standard definition, but it may provide a useful visual bridge.

---

⚠️ Important limitation: Rarity depends on the model

Just like Shannon surprisal, Information Rarity is not intrinsic to the observation.

For the same observation x:

Model A:
p(x) = 1/2
R(x) = 2

Model B:
p(x) = 1/100
R(x) = 100

The datum is unchanged.

The expectation is different.

Therefore the Rarity changes.

This reinforces an important principle:

Information is relational.

It exists between observation and model.

---

🏵 Proposed project vocabulary

For internal notes:

Probability
p(x)

Information Rarity
R(x) = 1/p(x)

Surprisal / Self-information
I(x) = log₂R(x)

Entropy
H(X) = E[I(X)]

This produces the sequence:

[ \boxed{ p(x) \rightarrow R(x) \rightarrow I(x) \rightarrow H(X) } ]

with:

[ \boxed{ p \rightarrow \frac1p \rightarrow \log_2\frac1p \rightarrow \mathbb E\left[\log_2\frac1p\right] } ]

---

🏵 What this chapter should accomplish

This section is useful if it helps the reader understand three things:

1. why rare observations produce large information
2. why the reciprocal 1/p is hiding inside Shannon's equation
3. why the logarithm converts multiplicative rarity into additive information

The target insight is:

Shannon information is not simply “negative log probability.”

It is the logarithmic scale of inverse probability.

Or, using the working vocabulary:

[ \boxed{ \text{Information}

\log_2(\text{Information Rarity}) } ]

---

🪄 Questions to explore later

The concept may deserve further investigation before becoming part of the main text.

Questions:

Can Information Rarity be represented geometrically?

Is there a useful network interpretation that remains mathematically exact?

Does conditional probability produce a useful notion of conditional Rarity?

How does Rarity behave under Bayesian updating?

Can likelihood ratios be interpreted as ratios of Rarity?

Can odds provide a better circuit analogy than raw probability?

Is there a useful relation to energy-like interpretations of -log p?

Does the analogy illuminate cross-entropy or KL divergence?

Would "Rarity", "Reticence", or another term produce less confusion?

These should remain open research/design questions rather than asserted theory.

---

☯ Working conclusion

Information Rarity is not a new information-theoretic quantity in the formal sense.

It is the familiar reciprocal:

[ R(x)=\frac1{p(x)} ]

given a pedagogical role.

Its value lies in exposing the hidden architecture:

Probability
    ↓ reciprocal

Rarity
    ↓ logarithm

Information
    ↓ expectation

Entropy

The important mathematical fact is:

[ \boxed{ I(x)

-\log_2p(x)

\log_2\frac1{p(x)}

\log_2R(x) } ]

The important conceptual fact is:

Probability tells us how open a path was.
Rarity tells us how narrow it was.
Information tells us, logarithmically, how much that narrowness mattered.