Testing - Toxic Berries

[bordumb]

Comprehensive ARLA Causal Reasoning Validation Plan

Statistical Analysis Framework (Applied to All Experiments)

Sample Size and Power Analysis

• N = 50 simulation runs per experimental condition
• Power analysis validation using G*Power software to confirm 80% power for detecting medium effect sizes (Cohen's d = 0.5)
• Alpha level = 0.05 for all statistical tests

Statistical Testing Protocol

1. Distribution Testing:

   • Shapiro-Wilk test for normality (α = 0.05)
   • Levene's test for homogeneity of variances (α = 0.05)

2. Primary Analysis:

   • If assumptions met: One-way ANOVA followed by Tukey's HSD post-hoc tests
   • If normality violated: Kruskal-Wallis test followed by Dunn's post-hoc tests
   • If variance equality violated: Welch's ANOVA followed by Games-Howell post-hoc tests

3. Multiple Comparisons Correction:

   • Holm-Bonferroni correction applied across all experiments to control family-wise error rate

4. Effect Size Calculation:

   • Cohen's d for all pairwise comparisons
   • Eta-squared (η²) for overall ANOVA effects

---

Experiment 1: Isolating Causal Reasoning from Sensory Architecture

Hypothesis

The CausalGraphSystem provides significant performance advantages beyond those attributable to sensory architecture or exploration strategies.

Experimental Groups

Group A: Heuristic Baseline (N=50)

• Agent Type: Baseline-Heuristic-Agent
• Cognitive Systems: None (rule-based behavior)
• Sensory Input: Direct access to environment state
• Decision Making: Move toward closest visible berry
• Exploration: Deterministic (no randomness)

Group B: Full Causal Model (N=50)

• Agent Type: Causal-QLearning-Agent
• Cognitive Systems: QLearningSystem + CausalGraphSystem + PerceptionComponent
• Sensory Input: Limited perception within vision range
• Decision Making: Q-learning with causal feedback
• Exploration: Epsilon-greedy (ε = 0.1, decay = 0.995)

Group C: Perception-Only Control (N=50)

• Agent Type: Perception-QLearning-Agent
• Cognitive Systems: QLearningSystem + PerceptionComponent (CausalGraphSystem disabled)
• Sensory Input: Identical to Group B
• Decision Making: Standard Q-learning without causal feedback
• Exploration: Identical epsilon-greedy parameters to Group B

Group D: Exploration-Matched Heuristic (N=50)

• Agent Type: Heuristic-Agent-with-Exploration
• Cognitive Systems: None (rule-based with random component)
• Sensory Input: Direct access to environment state
• Decision Making: 90% move toward closest berry, 10% random movement
• Exploration: Matched to approximate Q-learning exploration frequency

Environment Configuration

• Grid Size: 50x50 cells
• Berry spawn rates: Red (20%), Blue (15%), Yellow (15%)
• Water sources: 8-10 randomly placed
• Rock formations: 15-20 randomly placed
• Phase 1 (ticks 0-1000): Blue berries always safe
• Phase 2 (ticks 1000-1600): Blue berries toxic when within 2 tiles of water

Measurement Protocol

Primary Metrics:

1. Causal Understanding Score: Proportion of correct decisions regarding blue berries in novel contexts (ticks 1000-1100)
2. Average Agent Health: Mean health across population during adaptation period (ticks 1000-1200)

Process Metrics:
3. Causal Model Accuracy (Groups B only): Dowhy validation of learned causal relationships against ground truth
4. Behavioral Adaptation Speed: Number of ticks required to achieve 80% accuracy on blue berry decisions after environmental change

Control Metrics:
5. Exploration Coverage: Percentage of map tiles visited by each group
6. Berry Consumption Distribution: Proportion of each berry type consumed pre/post environmental change

Success Criteria

• Group B must significantly outperform Groups C and D on primary metrics (p < 0.05, d > 0.5)
• Group B must show superior causal model accuracy compared to baseline chance
• Performance differences must persist after controlling for exploration coverage

---

Experiment 2: Testing Genuine Causal Understanding vs. Simple Avoidance

Hypothesis

Agents with causal reasoning learn flexible, context-dependent rules rather than simple avoidance heuristics.

Experimental Groups

• Group B: Full Causal Model (from Experiment 1)
• Group C: Perception-Only Control (from Experiment 1)

Environment Modifications

Base Environment: Same as Experiment 1

Added Elements:

• Purifier Crystals: 5-7 visible crystal objects randomly placed on map
• Purification Rule: Blue berries within 2 tiles of crystals are always safe, overriding water toxicity
• Crystal Visibility: Crystals appear as distinct visual elements agents can perceive

Experimental Timeline

• Phase 1 (ticks 0-1000): Blue berries safe, crystals present but inactive
• Phase 2 (ticks 1000-1200): Blue berries toxic near water, crystals activate purification
• Phase 3 (ticks 1200-1600): Extended testing period

Measurement Protocol

Primary Metrics:

1. Safe Zone Consumption Rate: Number of blue berries consumed near crystals during Phase 2
2. Crystal Approach Behavior: Frequency of movement toward crystals when blue berries are visible nearby

Secondary Metrics:
3. Causal Chain Recognition: Behavioral evidence of understanding crystal→safety→blue berry relationship
4. Context Switching Accuracy: Correct identification of safe vs. unsafe blue berries based on environmental context

Process Validation:
5. Counterfactual Reasoning Test: Post-training query of causal models about crystal effects

Success Criteria

• Group B must show significantly higher Safe Zone Consumption Rate than Group C
• Group B must demonstrate Crystal Approach Behavior while Group C shows avoidance
• Process validation must confirm Group B learned crystal purification rule

---

Experiment 3: Temporal Causal Reasoning

Hypothesis

The CausalGraphSystem can identify and utilize temporal causal relationships involving delayed effects.

Experimental Groups

• Group B: Full Causal Model
• Group C: Perception-Only Control
• Both groups receive identical state information including explicit "metabolic boost" status flag

Environment Design

New Berry Type: Orange berries (10% spawn rate)
Temporal Causal Rule:

• Eating orange berry triggers 50-tick "metabolic boost" state
• During metabolic boost, all berry types provide double health benefits
• Metabolic boost status visible to agents in state representation

Experimental Timeline

• Training Phase (ticks 0-800): Agents learn basic environment
• Testing Phase (ticks 800-1600): Temporal causation active

Measurement Protocol

Primary Metrics:

1. Temporal Strategy Score: Frequency of seeking orange berries when health is low (indicating understanding of delayed benefit)
2. Optimal Timing Behavior: Rate of delaying valuable berry consumption until after eating orange berries

Secondary Metrics:
3. Boost Utilization Rate: Proportion of metabolic boost duration spent consuming high-value berries
4. Causal Discovery Speed: Ticks required to establish orange berry seeking behavior

Success Criteria

• Group B must show significantly higher Temporal Strategy Score than Group C
• Group B must demonstrate Optimal Timing Behavior patterns
• Behavioral differences must emerge despite identical state information access

---

Experiment 4: Learning Convergence Control

Hypothesis

Performance differences between causal and perception-only agents reflect cognitive architectural advantages, not differential learning time requirements.

Experimental Groups

Group C-Standard: Perception-Only agents run for standard 1600 ticks

Group C-Extended: Perception-Only agents run for 4800 ticks (3x duration)

Group C-Plateau: Perception-Only agents run until performance plateaus (no improvement for 200 consecutive ticks)

Convergence Detection Protocol

1. Calculate rolling 50-tick average of causal understanding score
2. Detect plateau when rolling average changes < 0.01 for 200 ticks
3. Compare final plateau performance against Group B performance at various time points

Measurement Protocol

Primary Metrics:

1. Plateau Performance Level: Final causal understanding score after convergence
2. Learning Efficiency: Ticks required to reach 90% of final performance level

Convergence Analysis:
3. Asymptotic Performance Comparison: Statistical comparison of final performance levels
4. Learning Curve Analysis: Comparison of learning trajectories using curve fitting

Success Criteria

• No significant difference between Group C-Standard and Group C-Extended final performance
• Group C-Plateau performance remains significantly below Group B performance
• Learning curve analysis confirms different asymptotes rather than different learning rates

---

Experiment 5: Direct Causal Model Validation

Hypothesis

Agents with CausalGraphSystem develop accurate internal representations of environmental causal structure.

Experimental Protocol

Model Extraction:

def extract_causal_model(agent_id, causal_system):
    """Extract learned causal relationships from agent's internal model"""
    return causal_system.get_causal_graph(agent_id)

def validate_causal_relationships(learned_model, ground_truth_rules):
    """Compare learned model against known environmental causation"""
    accuracy_scores = []
    
    for relationship in ground_truth_rules:
        predicted_effect = learned_model.estimate_effect(
            treatment=relationship.cause,
            outcome=relationship.effect,
            treatment_value=relationship.intervention_value
        )
        
        true_effect = relationship.ground_truth_effect
        accuracy = calculate_prediction_accuracy(predicted_effect, true_effect)
        accuracy_scores.append(accuracy)
    
    return np.mean(accuracy_scores)

Counterfactual Testing:
Test agents' ability to predict outcomes of hypothetical interventions:

1. "What would happen if blue berries appeared near crystals instead of water?"
2. "What would happen if orange berries lasted 100 ticks instead of 50?"
3. "What would happen if yellow berries never appeared near rocks?"

Measurement Protocol

Model Quality Metrics:

1. Causal Relationship Accuracy: Proportion of correctly identified causal links
2. Counterfactual Prediction Accuracy: Accuracy of hypothetical scenario predictions
3. Model Complexity Score: Number of causal relationships inferred (penalize overfitting)

Success Criteria

• Group B must achieve >80% accuracy on causal relationship identification
• Counterfactual predictions must significantly exceed chance performance
• Model complexity should be appropriate (not overfitted or underfitted)

---

Experiment 6: Cross-Domain Transfer Learning

Hypothesis

Causal reasoning capabilities generalize across different environmental domains and causal structures.

Transfer Environment: Tool Crafting Domain

Environment Design:

• Agents must collect resources and craft tools in sequence
• Multi-step causal chains: Wood + Stone → Hammer; Hammer + Metal → Sword
• Success requires understanding prerequisite relationships

Transfer Protocol:

1. Train agents in berry environment (800 ticks)
2. Transfer to tool crafting environment (no additional training)
3. Measure adaptation speed and final performance

Measurement Protocol

Transfer Metrics:

1. Adaptation Speed: Ticks required to achieve first successful tool craft
2. Final Performance: Tool crafting success rate in final 200 ticks
3. Causal Transfer Evidence: Behavioral indicators of applying causal reasoning to new domain

Success Criteria

• Group B must show faster adaptation speed than Group C
• Final performance differences must parallel original domain advantages
• Process analysis must confirm application of causal reasoning principles

---

Enhanced Exploration Controls

Standardized Exploration Protocol

All Q-learning agents use identical parameters:

• Initial epsilon: 0.1
• Epsilon decay rate: 0.995 per tick
• Minimum epsilon: 0.01
• Learning rate: 0.1
• Discount factor: 0.95

Exploration Validation Metrics

1. Map Coverage: Percentage of environment tiles visited
2. Action Diversity: Entropy of action selection distribution
3. Exploration Efficiency: Ratio of novel states discovered to total actions taken

---

Comprehensive Success Framework

Evidence FOR Causal Reasoning

• Significant behavioral advantages (d > 0.5) across multiple experiments
• High causal model accuracy in process validation (>80%)
• Successful transfer to novel domains
• Advantages persist after controlling for exploration and learning time
• Counterfactual reasoning capabilities exceed chance performance

Evidence AGAINST Causal Reasoning

• Performance differences eliminated by exploration/learning controls
• No advantage in process measures of causal model quality
• Poor transfer to new domains
• Advantages explained by confounding factors
• Random-level performance on counterfactual reasoning tests

Implementation Timeline

Weeks 1-2: Implement all experimental conditions and enhanced measurement systems
Weeks 3-4: Execute Experiments 1-3 with full statistical protocols
Weeks 5-6: Execute Experiments 4-6 and cross-experiment validation analyses
Week 7: Comprehensive analysis, replication checks, and results interpretation

[bordumb]

Root Cause Analysis of Q-Learning Agent Underperformance in the Berry World Simulation

1.0 Executive Summary

This report provides a comprehensive root cause analysis of the unexpected and dramatic underperformance of a Q-learning agent (the "Causal Agent") when compared to a simple heuristic agent (the "Baseline Agent") in the Berry World simulation. The A/B test was designed to demonstrate the superiority of a learned policy over a hardcoded one; however, the results showed the opposite.

The core finding of this analysis is that the Causal Agent's failure is not attributable to a flaw in the Q-learning algorithm, the neural network architecture, or its capacity to learn. Instead, the failure is a direct and predictable consequence of a critical deficiency in its sensory input pipeline: the state representation. The agent has been rendered effectively blind to the very objects it is tasked with finding. Its BerryStateEncoder provides it with information about its own internal status—position and health—but critically omits all information about the external world relevant to its survival, namely the location, distance, and type of berries. The agent is being asked to learn an optimal policy for a task it cannot perceive.

Consequently, the A/B test, as currently structured, is not a valid comparison of a "simple" versus a "complex" decision-making policy. It is an invalid experiment that compares an agent with privileged, oracle-like access to the environment's ground truth (the Baseline Agent) against an agent with almost no relevant sensory information (the Causal Agent). The outcome was predetermined by this severe information asymmetry, which invalidates any conclusions about the relative intelligence of the two approaches.

The immediate path forward is to redesign the BerryStateEncoder to provide the Q-learning agent with the necessary environmental features to make the learning task solvable. This involves augmenting the state vector with sensory information about nearby berries. A long-term recommendation is to revise the methodology for creating baseline agents to ensure they operate under the same informational constraints as the learning agents, thereby creating fair, meaningful, and scientifically valid comparisons.

2.0 Analysis of Experimental Results: A Quantitative View of Failure

A thorough examination of the final experimental data reveals a clear and consistent narrative of the Causal Agent's failure. The quantitative metrics do not merely show underperformance; they paint a detailed picture of an agent engaging in a fruitless, random search, contrasted with a baseline agent executing a directed, purposeful strategy.

Average Total Actions Per Run

The most immediate indicator of behavioral difference is the total activity level of each agent type. The Baseline Agent performed an average of 779.00 actions per run, while the Causal Agent performed only 573.33. This discrepancy of over 200 actions is significant. In a foraging task, a higher action count is a primary indicator of purposeful behavior. An agent that knows where to go, or at least has a strategy for finding its objective, will spend less time idle and take more concrete steps toward its goal. The Baseline Agent's higher activity level reflects its ability to consistently identify a target (the closest berry) and execute a plan to reach it. Conversely, the Causal Agent's lower action count suggests a pattern of indecisiveness, prolonged inaction, or inefficient wandering—behaviors that are entirely consistent with an agent that lacks the necessary information to form a coherent plan. Without a clear objective in its sensory input, the agent has no impetus to act decisively, leading to fewer actions taken over the course of the simulation.

Average Eat Actions and Percentage of Eat Actions

At first glance, the avg_eat_actions_per_run metric appears contradictory: the Causal Agent (38.00) performed significantly more "eat" actions than the successful Baseline Agent (12.67). However, when viewed in conjunction with the pct_eat_berry metric, this is revealed not as a sign of success, but as a symptom of inefficient and desperate behavior.

The pct_eat_berry metric is the most damning piece of evidence in the dataset. For the Baseline Agent, 2.48% of its total actions resulted in successfully eating a berry. For the Causal Agent, this figure was a mere 0.30%. This means the Baseline Agent's actions were over eight times more likely to be productive in terms of nourishment. This metric starkly quantifies the difference between directed behavior and a random search. The Baseline Agent's high percentage indicates that its move actions are highly correlated with berry locations, successfully taking it from one food source to the next. The Causal Agent's near-zero percentage confirms that its movements are almost entirely uncorrelated with berry locations.

This context reframes the avg_eat_actions data. The Causal Agent is not eating more because it is a better forager; it is eating more frequently because it is stumbling upon berries by chance during its random walk and consuming them immediately out of necessity. Since it cannot perceive other berries in the environment, it has no capacity for strategic planning. Any berry it finds is consumed, regardless of its current health, because it has no information to suggest another, better opportunity exists. The Baseline Agent, in contrast, can see all nearby berries and can therefore execute a more efficient foraging plan. It might ignore a nearby berry in favor of a cluster of berries further away, leading to fewer individual "eat" actions but a much higher overall rate of resource collection relative to actions taken. The Causal Agent is engaging in random, opportunistic snacking; the Baseline Agent is foraging with intent.

Percentage of Move Actions

The final metric, pct_move, shows that both agents spend the vast majority of their time moving: 97.52% for the Baseline Agent and 99.70% for the Causal Agent. This is expected in a spatial foraging simulation. The slightly higher percentage for the Causal Agent reinforces the narrative of fruitless wandering. Because it so rarely finds itself in a position to eat a berry (as shown by its low pct_eat_berry), it has very few opportunities to do anything but move. Its action space is effectively reduced to random movement because it lacks the sensory input to make any other action, like a targeted move or an eat, a viable option.

3.0 Deconstruction of Agent Architectures: Information is Power

The quantitative results are a direct consequence of a fundamental disparity in the information available to each agent. A line-by-line analysis of the provided code reveals that the two agents operate under vastly different perceptual models. One is an omniscient oracle, while the other is a blind learner. Their performance is a direct function of the quality of information they receive, not the sophistication of their decision-making logic.

3.1 The Baseline Agent: An Oracle-Guided Heuristic

The success of the Baseline Agent is not due to a particularly clever heuristic, but rather to the fact that it is granted privileged, direct access to the simulation's ground truth. Its "brain," implemented in the BerryDecisionSelector class, functions as an oracle.

The critical code block that grants this power is as follows :

Python

# FILE: simulations/berry_sim/providers.py
pos_comp = sim_state.get_component(entity_id, PositionComponent)

env = sim_state.environment

if pos_comp and isinstance(env, BerryWorldEnvironment) and move_actions:

#...

# This agent can "see" all berries in the environment

for berry_pos in env.berry_locations.keys():

dist = env.distance(pos_comp.position, berry_pos)

if dist < min_dist and dist <= vision_range:

min_dist = dist

closest_berry_pos = berry_pos

<textarea data-mprt="7" class="inputarea monaco-mouse-cursor-text" wrap="off" autocorrect="off" autocapitalize="off" autocomplete="off" spellcheck="false" aria-label="Editor content" aria-required="false" tabindex="0" role="textbox" aria-roledescription="editor" aria-multiline="true" aria-autocomplete="both" style="tab-size: 33.609375px; font-family: "Google Sans Mono", Menlo, Monaco, "Courier New", monospace; font-weight: normal; font-size: 14px; font-feature-settings: "liga" 0, "calt" 0; font-variation-settings: normal; line-height: 18px; letter-spacing: 0px; top: 0px; left: 0px; width: 1px; height: 18px;"></textarea>

The line for berry_pos in env.berry_locations.keys(): is the source of this privileged access. The agent is not sensing its environment; it is directly querying the environment's master list of all berry locations. This is information that a real-world, embodied agent would never possess. It can then perfectly calculate the distance to every single berry and identify the absolute closest one.

This "Oracle Problem" is a common but methodologically flawed approach to creating a baseline. The agent is not "simple" in an informational sense; it is omniscient within its local area. Its success is preordained by being given the answer to the problem it is supposed to be solving. This makes it an unfair and uninformative benchmark for comparison against a learning agent that is expected to discover a strategy from limited sensory data.

3.2 The Causal Agent: A Blind Learner

In stark contrast, the Causal Agent adheres to a more realistic and architecturally sound model of perception, but it is crippled by the poverty of the information it receives. Its decision-making process, implemented in QLearningDecisionSelector, is entirely dependent on the output of a state encoder :

Python

# FILE: simulations/berry_sim/providers.py

# The agent's entire world view comes from this state_features vector

state_features = self.state_encoder.encode_state(sim_state, entity_id, self.config)

state_tensor = torch.tensor(state_features, dtype=torch.float32).unsqueeze(0)

<textarea data-mprt="7" class="inputarea monaco-mouse-cursor-text" wrap="off" autocorrect="off" autocapitalize="off" autocomplete="off" spellcheck="false" aria-label="Editor content" aria-required="false" tabindex="0" role="textbox" aria-roledescription="editor" aria-multiline="true" aria-autocomplete="both" style="tab-size: 33.609375px; font-family: "Google Sans Mono", Menlo, Monaco, "Courier New", monospace; font-weight: normal; font-size: 14px; font-feature-settings: "liga" 0, "calt" 0; font-variation-settings: normal; line-height: 18px; letter-spacing: 0px; top: 0px; left: 0px; width: 1px; height: 18px;"></textarea>

The agent's neural network has no other channel through which to perceive the world. Its ability to learn is therefore fundamentally constrained by the quality of the state_features vector. An analysis of the BerryStateEncoder class reveals this to be the root cause of failure :

Python

# FILE: simulations/berry_sim/providers.py

class BerryStateEncoder(StateEncoderInterface):

def encode_state(self, sim_state: Any, entity_id: str, config: Any,...) -> np.ndarray:

#... (normalize x, y, health)...

agent_x = pos_comp.x / width if pos_comp else 0.5

agent_y = pos_comp.y / height if pos_comp else 0.5

health = health_comp.current_health / health_comp.initial_health if health_comp else 0.5
    <span class="hljs-comment"># The entire feature vector only contains the agent's own position and health.</span>
    <span class="hljs-comment"># It has no information about berries in the environment.</span>
    <span class="hljs-keyword">return</span> np.array([agent_x, agent_y, health], dtype=np.float32)

<textarea data-mprt="7" class="inputarea monaco-mouse-cursor-text" wrap="off" autocorrect="off" autocapitalize="off" autocomplete="off" spellcheck="false" aria-label="Editor content" aria-required="false" tabindex="0" role="textbox" aria-roledescription="editor" aria-multiline="true" aria-autocomplete="both" style="tab-size: 33.609375px; font-family: "Google Sans Mono", Menlo, Monaco, "Courier New", monospace; font-weight: normal; font-size: 14px; font-feature-settings: "liga" 0, "calt" 0; font-variation-settings: normal; line-height: 18px; letter-spacing: 0px; top: 0px; left: 0px; width: 1px; height: 18px;"></textarea>

The final feature vector returned by this method contains only three values: the agent's own normalized x-position, y-position, and health. There is a complete and total absence of information regarding the location, distance, direction, or type of berries. This is a classic "Garbage In, Garbage Out" (GIGO) problem. The Q-learning algorithm and its underlying neural network are functioning as designed, but they are being fed a state representation that is functionally useless for the task at hand. The agent is being asked to learn a mapping from its state and actions to expected rewards, but its state representation lacks the very features—the location of berries—that are predictive of reward.

4.0 The Core Flaw: Violation of Observability in Reinforcement Learning

The failure of the Causal Agent can be understood not just as a code-level bug, but as a fundamental violation of the theoretical principles that underpin reinforcement learning. The flawed state representation makes the learning problem mathematically unsolvable for the given algorithm.

4.1 The Markov Property: The Foundation of RL

Modern reinforcement learning is built upon the mathematical framework of the Markov Decision Process (MDP). A central assumption of an MDP is that the state signal, St​, is Markovian. This means the state at time t contains all necessary information from the past to fully determine the future. Formally, the probability of transitioning to the next state St+1​ given the current state St​ and action At​ is independent of all previous states and actions:

P(St+1​∣St​,At​)=P(St+1​∣St​,At​,St−1​,At−1​,…,S0​,A0​)

The state representation provided to the Causal Agent, s=(x,y,health), flagrantly violates this property for the berry-eating task. The agent's position and health alone are insufficient to predict the outcome of its actions.

To make this violation concrete, consider a simple thought experiment.

• Scenario A: The agent is at position (10,10) with full health. A rewarding red berry is at position (11,10). The optimal action is to move East.

• Scenario B: The agent is at position (10,10) with full health. A rewarding red berry is at position (9,10). The optimal action is to move West.

In both scenarios, the agent's state representation as defined by BerryStateEncoder is identical: sA​=sB​=(x=0.2,y=0.2,health=1.0) (assuming a 50x50 grid). The Q-learning algorithm receives the exact same input vector for two distinct world configurations that demand opposite optimal actions. The algorithm's goal is to learn a Q-function, Q(s,a), that predicts the expected future reward for taking action a in state s. Because sA​=sB​, the network must learn a single value for Q(s,"move East") and a single value for Q(s,"move West"). Over many trials, it will experience positive rewards for moving East in situations like Scenario A and negative (or zero) rewards for moving East in situations like Scenario B. The learning algorithm will be forced to average these conflicting outcomes, converging on Q-values that are not meaningfully different for any direction. This makes it mathematically impossible for the agent to learn a coherent policy, and its behavior will be no better than random guessing.

4.2 Operating in a POMDP Without the Right Tools

When the Markov property is violated, the agent is no longer in an MDP. It is operating in a Partially Observable Markov Decision Process (POMDP). In a POMDP, the agent does not perceive the true state of the world, s, but instead receives an observation, o, which provides only partial information about s.

Solving POMDPs is significantly more challenging than solving MDPs. Standard feed-forward Q-networks, like the one used by the Causal Agent, are memoryless and perform poorly in such conditions. To succeed, an agent typically needs to be equipped with memory, for instance by using a Recurrent Neural Network (RNN) architecture like an LSTM. This memory allows the agent to integrate observations over time to build an internal "belief state" that better approximates the true, unobserved state of the world (e.g., by creating a mental map of where it has seen berries). The Causal Agent was effectively placed in a POMDP but was only given the tools to solve an MDP. Its failure was therefore predictable from a theoretical standpoint.

5.0 Invalidating the A/B Test: An Asymmetric Information Problem

The issues identified thus far go beyond a simple implementation error; they represent a fundamental flaw in the experimental design of the A/B test. The comparison between the Baseline and Causal agents is not a meaningful test of "heuristic vs. learned intelligence" because the two agents are not playing the same game. The severe information asymmetry invalidates the experiment.

5.1 Violation of Architectural Principles

The supplementary research material details the Affective Reflective Learning Architecture (ARLA), the platform on which this simulation is built. The documentation makes it clear that ARLA is designed with a strict three-layer separation of concerns: agent-core (the "Soul"), agent-engine (the "Brain"), and the final simulation application (the "Body" and "World"). A key feature of this design is the use of ProviderInterface classes, such as StateEncoderInterface, which act as the sole, contractually-defined bridge for passing information from the world to the agent's brain.

The berry_sim experiment inadvertently subverts this sound architectural design.

• The Causal Agent correctly adheres to the architecture. Its brain (QLearningDecisionSelector) receives all its information through the designated bridge, the StateEncoderInterface. However, it is punished for this adherence because the concrete implementation of that bridge (BerryStateEncoder) is faulty.

• The Baseline Agent violates the architecture. Its brain (BerryDecisionSelector) bypasses the designated informational bridge and reaches directly into the simulation's world state (env.berry_locations) to gain privileged knowledge.

The experiment, therefore, does not test the agents' intelligence. It tests their adherence to the system's architecture and accidentally rewards the agent that cheats.

5.2 Information Asymmetry Analysis

The core flaw of the experiment can be summarized by analyzing the disparity in information available to each agent. The following table provides a stark, at-a-glance summary of this asymmetry, making the invalidity of the comparison undeniable.

Information Feature	Baseline Agent (Heuristic)	Causal Agent (Q-Learning)
Agent's Own Position (x, y)	✅ Yes	✅ Yes
Agent's Own Health	❌ No (Implicitly used)	✅ Yes
Location of ALL Berries	✅ Yes (Privileged Access)	❌ No
Distance to Closest Berry	✅ Yes (Calculated from All)	❌ No
Type of Closest Berry	✅ Yes (Implicitly known)	❌ No

Export to Sheets

This table makes it clear that the Causal Agent is information-starved, while the Baseline Agent is information-rich. The test is fundamentally unfair, and its results are therefore meaningless for evaluating the efficacy of Q-learning in this environment.

6.0 Recommendations for a Valid Experimental Design

To rectify this situation and enable a valid comparison of decision-making strategies, both the Causal Agent's sensory system and the experimental methodology must be revised. The following recommendations provide a clear path toward a robust and scientifically sound A/B test.

6.1 Primary Recommendation: Redesigning the State Encoder

The most critical and immediate change required is to redesign the BerryStateEncoder to provide the Causal Agent with the sensory information it needs to solve the task. The state representation must be augmented to describe the agent's local perception of the environment, making the state Markovian for the foraging task. A common and effective approach for representing perceived objects in an agent's vicinity is to use polar coordinates (distance and angle) relative to the agent.

The following high-level code demonstrates a revised BerryStateEncoder. This implementation assumes a PerceptionComponent has been added to the agent, which is a standard pattern in ECS architectures and is already defined within agent-core. This component would be populated by a world-specific PerceptionProvider that identifies entities within the agent's vision range.

Python

# FILE: simulations/berry_sim/providers.py
import math

import numpy as np

from agent_core.policy.state_encoder_interface import StateEncoderInterface

from agent_core.core.ecs.component import PerceptionComponent, PositionComponent, HealthComponent
class BerryStateEncoder(StateEncoderInterface):

def encode_state(self, sim_state: Any, entity_id: str, config: Any,...) -> np.ndarray:

"""

Encodes the agent's internal state and its local perception of berries.

"""

# --- 1. Encode Agent's Internal State ---

pos_comp = sim_state.get_component(entity_id, PositionComponent)

health_comp = sim_state.get_component(entity_id, HealthComponent)

width = config.environment.params.width

height = config.environment.params.height
    agent_x = pos_comp.x / width <span class="hljs-keyword">if</span> pos_comp <span class="hljs-keyword">else</span> <span class="hljs-number">0.5</span>
    agent_y = pos_comp.y / height <span class="hljs-keyword">if</span> pos_comp <span class="hljs-keyword">else</span> <span class="hljs-number">0.5</span>
    health = health_comp.current_health / health_comp.initial_health <span class="hljs-keyword">if</span> health_comp <span class="hljs-keyword">else</span> <span class="hljs-number">0.5</span>
    agent_state_vector = [agent_x, agent_y, health]

    <span class="hljs-comment"># --- 2. Encode Agent's Perception of the World ---</span>
    perc_comp = sim_state.get_component(entity_id, PerceptionComponent)
    
    <span class="hljs-comment"># Initialize features for nearest red, blue, and yellow berries.</span>
    <span class="hljs-comment"># Each feature is a tuple of (normalized distance, normalized angle).</span>
    <span class="hljs-comment"># Default to (1.0, 0.0) indicating "no berry seen at max distance, forward direction".</span>
    perception_features = [<span class="hljs-number">1.0</span>, <span class="hljs-number">0.0</span>] * <span class="hljs-number">3</span>  <span class="hljs-comment"># 3 berry types * 2 features each</span>

    <span class="hljs-keyword">if</span> perc_comp <span class="hljs-keyword">and</span> pos_comp:
        <span class="hljs-comment"># Find the closest of each type of visible berry from the perception component</span>
        nearest_berries = {<span class="hljs-string">"red"</span>: <span class="hljs-literal">None</span>, <span class="hljs-string">"blue"</span>: <span class="hljs-literal">None</span>, <span class="hljs-string">"yellow"</span>: <span class="hljs-literal">None</span>}
        <span class="hljs-keyword">for</span> entity_data <span class="hljs-keyword">in</span> perc_comp.visible_entities.values():
            <span class="hljs-keyword">if</span> entity_data.get(<span class="hljs-string">"type"</span>) == <span class="hljs-string">"berry"</span>:
                b_type = entity_data.get(<span class="hljs-string">"berry_type"</span>)
                <span class="hljs-keyword">if</span> b_type <span class="hljs-keyword">in</span> nearest_berries:
                    <span class="hljs-keyword">if</span> nearest_berries[b_type] <span class="hljs-keyword">is</span> <span class="hljs-literal">None</span> <span class="hljs-keyword">or</span> entity_data[<span class="hljs-string">"distance"</span>] &lt; nearest_berries[b_type][<span class="hljs-string">"distance"</span>]:
                        nearest_berries[b_type] = entity_data

        <span class="hljs-comment"># Encode distance and angle for each berry type</span>
        feature_idx = <span class="hljs-number">0</span>
        <span class="hljs-keyword">for</span> b_type <span class="hljs-keyword">in</span> [<span class="hljs-string">"red"</span>, <span class="hljs-string">"blue"</span>, <span class="hljs-string">"yellow"</span>]:
            berry_data = nearest_berries[b_type]
            <span class="hljs-keyword">if</span> berry_data:
                <span class="hljs-comment"># Normalize distance by vision range</span>
                dist = berry_data[<span class="hljs-string">"distance"</span>] / perc_comp.vision_range
                
                <span class="hljs-comment"># Calculate angle relative to agent and normalize to [-1, 1]</span>
                berry_pos = berry_data[<span class="hljs-string">"position"</span>]
                dx = berry_pos - pos_comp.x
                dy = berry_pos - pos_comp.y
                angle = math.atan2(dy, dx) / math.pi
                
                perception_features[feature_idx] = dist
                perception_features[feature_idx + <span class="hljs-number">1</span>] = angle
            feature_idx += <span class="hljs-number">2</span>

    <span class="hljs-comment"># --- 3. Combine and Return the Full State Vector ---</span>
    <span class="hljs-keyword">return</span> np.array(agent_state_vector + perception_features, dtype=np.float32)

<textarea data-mprt="7" class="inputarea monaco-mouse-cursor-text" wrap="off" autocorrect="off" autocapitalize="off" autocomplete="off" spellcheck="false" aria-label="Editor content" aria-required="false" tabindex="0" role="textbox" aria-roledescription="editor" aria-multiline="true" aria-autocomplete="both" style="tab-size: 33.609375px; font-family: "Google Sans Mono", Menlo, Monaco, "Courier New", monospace; font-weight: normal; font-size: 14px; font-feature-settings: "liga" 0, "calt" 0; font-variation-settings: normal; line-height: 18px; letter-spacing: 0px; top: 0px; left: 0px; width: 1px; height: 18px;"></textarea>

Justification: This new state representation, with a vector like [x, y, health, dist_red, angle_red, dist_blue, angle_blue, dist_yellow, angle_yellow], is now sufficiently informative. It gives the agent the necessary inputs (distance and direction to objectives) to learn a meaningful policy that can correlate its actions (e.g., moving in a certain direction) with desirable outcomes (getting closer to a berry and receiving a reward). This fixes the Markov property violation and makes the learning problem tractable.

6.2 Secondary Recommendation: Ensuring a Fair Baseline

Fixing the Causal Agent is only half the solution. To conduct a valid A/B test, the Baseline Agent must also be modified to operate under the same informational constraints. Continuing to allow it to function as an "oracle" would still result in an unfair comparison.

The goal of the experiment should be to isolate and test a single variable: the decision-making logic (i.e., a learned neural network policy versus a hardcoded heuristic policy). To achieve this, all other variables, especially the perceptual system, must be held constant.

Proposed Change: The BerryDecisionSelector for the Baseline Agent should be refactored. Instead of receiving the global sim_state and accessing env.berry_locations directly, it should be passed the exact same feature vector generated by the new, improved BerryStateEncoder. Its hardcoded logic would then operate on this vector. For example, its heuristic could be: "Examine the three distance features in the state vector; find the index corresponding to the minimum distance; and choose the move action that most directly reduces that distance."

By giving both agents the same sensory input, any resulting difference in performance can be correctly and confidently attributed to the quality of their respective policies. This is a cornerstone of rigorous scientific experimentation in artificial intelligence and is essential for generating trustworthy and meaningful results.