# Troubleshooting Guide

Common issues and their solutions when using Morphium.

## Connection Issues

### Cannot Connect to MongoDB

**Symptoms:**
- `MorphiumDriverException: Could not get connection to [host] in time [timeout]ms`
- Application startup fails with connection timeout

**Diagnosis:**
```xml
<!-- Enable debug logging for driver in log4j2.xml -->
<Logger name="de.caluga.morphium.driver.wire.PooledDriver" level="DEBUG" additivity="false">
    <AppenderRef ref="Console"/>
</Logger>
```

**Common Causes & Solutions:**

1. **Incorrect host/port configuration**
   ```java
   // Check your host configuration
   cfg.clusterSettings().addHostToSeed("correct-hostname", 27017);
   ```

2. **Network connectivity issues**
   - Verify MongoDB is running: `mongosh --host hostname:port`
   - Check firewall rules and network policies
   - Test connectivity: `telnet hostname 27017`

3. **Authentication failures**
   ```java
   // Verify credentials
   cfg.authSettings().setMongoLogin("correct-username");
   cfg.authSettings().setMongoPassword("correct-password");
   ```

4. **Connection pool exhaustion**
   ```java
   // Increase connection limits
   cfg.connectionSettings().setMaxConnectionsPerHost(200);
   cfg.connectionSettings().setMaxWaitTime(10000);
   ```

### Replica Set Connection Issues

**Symptoms:**
- `MorphiumDriverException: No primary node defined - not connected yet?`
- Intermittent connection failures

**Solutions:**
1. **Verify replica set configuration**
   ```javascript
   // In MongoDB shell
   rs.status()
   rs.conf()
   ```

2. **Check replica set name**
   ```java
   cfg.clusterSettings().setReplicaSetName("correct-replica-set-name");
   ```

3. **Add all replica set members to seed list**
   ```java
   cfg.clusterSettings().addHostToSeed("mongo1", 27017);
   cfg.clusterSettings().addHostToSeed("mongo2", 27017);
   cfg.clusterSettings().addHostToSeed("mongo3", 27017);
   ```

### Every write of one entity type takes exactly `wtimeout` after a node went down

**Symptom:** after a replica-set member crashed or was shut down, `store()` for one entity type (often a
test entity such as `UncachedObject`) takes exactly its `@WriteSafety.timeout` (10 s by default in the
test suite) on every call, while writes of other types are fast. The server logs
`Write concern timeout ... w=3, acknowledged=1/2` or similar; the client logs nothing, because the
write *did* succeed on the primary and the missed replication only arrives as a `writeConcernError`,
which Morphium does not surface yet (#408).

**Cause:** the entity uses `@WriteSafety(level = WAIT_FOR_ALL_SLAVES)`. Morphium turns that into a
numeric `w` equal to the number of replica-set members the driver can currently reach. Until the
driver's heartbeat has failed against the dead member once (at most one `heartbeatFrequency` plus the
connect timeout), `w` is still the full member count and cannot be satisfied.

**What to check:**
- The hang should end by itself after the detection window. If it does not, the driver still counts the
  dead member as reachable: enable debug logging for `de.caluga.morphium.Morphium` and watch
  `Active nodes now: N` on each write, and `Could not create connection to host` /
  `Still cannot connect to host` from the `PooledDriver` heartbeat.
- `heartbeatFrequency` (`clusterSettings().setHeartbeatFrequency(ms)`) bounds the window; a lower value
  shortens it at the cost of more hello traffic.
- `timeout = 0` on the annotation means "wait forever": writes inside the window never return. Use a
  finite `wtimeout`.
- If you cannot afford the window, switch the entity to `SafetyLevel.MAJORITY`, which the server
  evaluates against the real configuration and which tolerates a minority of members being down.

See [Write Safety](./developer-guide.md#write-safety-writesafety) in the developer guide for the
semantics of each level.

## Performance Issues

### Slow Query Performance

**Symptoms:**
- High response times
- CPU usage spikes
- Memory growth

**Diagnosis:**
```xml
<!-- Enable query logging in log4j2.xml -->
<Logger name="de.caluga.morphium" level="DEBUG" additivity="false">
    <AppenderRef ref="Console"/>
</Logger>
```

```java
// Monitor driver statistics
Map<DriverStatsKey, Double> stats = morphium.getDriver().getDriverStats();
System.out.println("Connections in use: " + stats.get(DriverStatsKey.CONNECTIONS_IN_USE));
System.out.println("Connections in pool: " + stats.get(DriverStatsKey.CONNECTIONS_IN_POOL));
```

**Solutions:**

1. **Add missing indexes**
   ```java
   @Entity
   @Index({"fieldName", "composite,field"}) // Add compound indexes
   public class MyEntity {
       @Index
       private String frequentlyQueriedField;
   }
   
   // Create indexes manually
   morphium.ensureIndicesFor(MyEntity.class);
   ```

2. **Optimize query patterns**
   ```java
   // Use projection to limit returned fields
   Query<MyEntity> q = morphium.createQueryFor(MyEntity.class)
       .f("status").eq("active")
       .project("name", "email"); // Only return specific fields
   
   // Use pagination for large result sets
   List<MyEntity> results = q.skip(offset).limit(pageSize).asList();
   ```

3. **Review caching configuration**
   ```java
   @Entity
   @Cache(timeout = 60000, maxEntries = 10000) // Adjust cache settings
   public class MyEntity { /* ... */ }
   ```

### Memory Issues

**Symptoms:**
- `OutOfMemoryError`
- Gradual memory increase
- Long GC pauses

**Diagnosis:**
```java
// Monitor connection pool stats
Map<DriverStatsKey, Double> stats = morphium.getDriver().getDriverStats();
Double connectionsInUse = stats.get(DriverStatsKey.CONNECTIONS_IN_USE);
Double connectionsInPool = stats.get(DriverStatsKey.CONNECTIONS_IN_POOL);

System.out.println("Memory usage - Connections: " + connectionsInUse + "/" + connectionsInPool);
```

**Solutions:**

1. **Connection pool sizing**
   ```java
   // Adjust pool size based on actual usage
   cfg.connectionSettings().setMaxConnectionsPerHost(50); // Reduce if too high
   cfg.connectionSettings().setMaxConnectionIdleTime(60000); // Close idle connections sooner
   cfg.connectionSettings().setMaxConnectionLifetime(300000); // Shorter connection lifetime
   ```

2. **Cache tuning**
   ```java
   @Entity
   @Cache(maxEntries = 1000, timeout = 30000, strategy = Cache.ClearStrategy.LRU)
   public class MyEntity { /* ... */ }
   ```

3. **Large result set handling**
   ```java
   // Use iterator for large datasets
   MorphiumIterator<MyEntity> iterator = query.asIterable(100); // Process in batches
   while (iterator.hasNext()) {
       MyEntity entity = iterator.next();
       // Process entity
   }
   ```

## Messaging Issues

### Messages Not Being Processed

**Symptoms:**
- Messages accumulating in queue
- Listeners not receiving messages
- High message latency

**Diagnosis:**
```java
// Check messaging statistics
MessagingStatsKey stats = messaging.getStats();
System.out.println("Messages processed: " + stats.getMessagesProcessed());
System.out.println("Messages failed: " + stats.getMessagesFailed());
```

**Solutions:**

1. **Check listener registration**
   ```java
   // Ensure listener is properly registered
   messaging.addListenerForTopic("mytopic", (morphiumMessaging, msg) -> {
       System.out.println("Processing: " + msg.getMsg());
       return null; // or return response message
   });
   
   // Verify messaging is started
   messaging.start();
   ```

2. **Increase processing capacity**
   ```java
   // Configure for higher throughput
   cfg.messagingSettings().setMessagingWindowSize(200);
   cfg.messagingSettings().setMessagingMultithreadded(true);
   cfg.messagingSettings().setPollPauseTime(100); // Reduce pause time
   ```

3. **Check for exceptions in listeners**
   ```java
   messaging.addListenerForTopic("mytopic", (morphiumMessaging, msg) -> {
       try {
           // Your processing logic
           return processMessage(msg);
       } catch (Exception e) {
           log.error("Message processing failed", e);
           // Decide whether to return error response or null
           return null;
       }
   });
   ```

### Message Ordering Issues

**Symptoms:**
- Messages processed out of order
- Race conditions in message processing

**Solutions:**

1. **Use exclusive messaging for ordering**
   ```java
   Msg msg = new Msg("ordered-topic", "content", "value");
   msg.setExclusive(true); // Ensure only one listener processes at a time
   messaging.sendMessage(msg);
   ```

2. **Single-threaded processing for critical topics**
   ```java
   // Configure single-threaded processing for specific topics
   cfg.messagingSettings().setMessagingMultithreadded(false);
   ```

## Logging and Debugging

### Enable Debug Logging

```xml
<!-- Configure logging in log4j2.xml -->
<Configuration>
    <Appenders>
        <Console name="Console">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
        </Console>
        <File name="MorphiumLog" fileName="/var/log/morphium/app.log">
            <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
        </File>
    </Appenders>
    <Loggers>
        <!-- Global debug logging for all Morphium components -->
        <Logger name="de.caluga.morphium" level="DEBUG" additivity="false">
            <AppenderRef ref="Console"/>
            <AppenderRef ref="MorphiumLog"/>
        </Logger>
        
        <!-- Class-specific logging -->
        <Logger name="de.caluga.morphium.driver.wire.PooledDriver" level="DEBUG" additivity="false">
            <AppenderRef ref="MorphiumLog"/>
        </Logger>
        <Logger name="de.caluga.morphium.messaging.SingleCollectionMessaging" level="DEBUG" additivity="false">
            <AppenderRef ref="MorphiumLog"/>
        </Logger>
        
        <!-- Package-level logging -->
        <Logger name="de.caluga.morphium.driver" level="DEBUG" additivity="false">
            <AppenderRef ref="MorphiumLog"/>
        </Logger>
    </Loggers>
</Configuration>
```

### Key Log Patterns to Watch

**Connection Issues:**
```
ERROR: Could not get connection to [host] in time [timeout]ms
WARN: Connection timeout
ERROR: Could not create connection to host [host]
```

**Performance Issues:**
```
WARN: Connection pool exhausted
DEBUG: Query took [time]ms
WARN: Cache hit ratio low: [ratio]
```

**Messaging Issues:**
```
ERROR: Message processing failed
WARN: Message queue size growing: [size]
DEBUG: Message processed in [time]ms
```

## Monitoring Key Metrics

### Connection Pool Metrics
```java
Map<DriverStatsKey, Double> stats = morphium.getDriver().getDriverStats();

// Monitor these key metrics:
Double connectionsInUse = stats.get(DriverStatsKey.CONNECTIONS_IN_USE);
Double connectionsInPool = stats.get(DriverStatsKey.CONNECTIONS_IN_POOL);
Double connectionsOpened = stats.get(DriverStatsKey.CONNECTIONS_OPENED);
Double connectionsClosed = stats.get(DriverStatsKey.CONNECTIONS_CLOSED);
Double errors = stats.get(DriverStatsKey.ERRORS);
Double threadsWaiting = stats.get(DriverStatsKey.THREADS_WAITING_FOR_CONNECTION);
```

### Performance Alerts

Set up monitoring for these conditions:

1. **Connection pool exhaustion**
   - `connectionsInUse / maxConnectionsPerHost > 0.8`

2. **High error rate**
   - `errors / (connectionsOpened + errors) > 0.05`

3. **Thread starvation**
   - `threadsWaiting > 0` for extended periods

4. **Connection churn**
   - High `connectionsOpened` and `connectionsClosed` rates

## Common Configuration Mistakes

### 1. Undersized Connection Pool
```java
// Problem: Too few connections for load
cfg.connectionSettings().setMaxConnectionsPerHost(5); // Too small

// Solution: Size based on concurrent operations
cfg.connectionSettings().setMaxConnectionsPerHost(50);
```

### 2. Excessive Connection Timeout
```java
// Problem: Long timeouts hide connection issues
cfg.connectionSettings().setMaxWaitTime(60000); // Too long

// Solution: Fail fast to detect issues
cfg.connectionSettings().setMaxWaitTime(5000);
```

### 3. Missing Indexes
```java
// Problem: No indexes on frequently queried fields
@Entity
public class User {
    private String email; // Frequently queried but not indexed
}

// Solution: Add appropriate indexes
@Entity
@Index({"email", "status,email"}) // Single and compound indexes
public class User {
    @Index
    private String email;
    private String status;
}
```

### 4. Inefficient Cache Configuration
```java
// Problem: Cache settings don't match usage patterns
@Cache(timeout = 1000, maxEntries = 100) // Too short timeout, too small cache

// Solution: Match cache to data access patterns
@Cache(timeout = 300000, maxEntries = 10000, strategy = Cache.ClearStrategy.LRU)
```

## PoppyDB-Specific Issues

### Illegal Opcode / Wire Protocol Corruption

**Symptoms:**
- `MorphiumDriverNetworkException: Illegal opcode 0` or random large opcode values
- `wrong section ID` errors

**Cause (fixed in 6.2.2):** Concurrent writes to the same Netty channel from different threads. Change stream event responses were written from background threads while the I/O thread was writing other responses.

**Solution:** Upgrade to 6.2.2+. If you're on an older version, reduce concurrent change stream watches per connection.

### Thread Leak (PooledDriver)

**Symptoms:**
- Thread count grows continuously over time
- `ConnectionWaiter` threads never terminate
- Eventually `OutOfMemoryError: unable to create new native thread`

**Cause (fixed in 6.2.2):** `PooledDriver.close()` did not signal waiting `ConnectionWaiter` threads. They remained blocked on `waitCounterCondition` forever.

**Solution:** Upgrade to 6.2.2+. As a workaround, ensure `Morphium.close()` is always called in a finally block.

### Change Stream Events Lost After Collection Drop

**Symptoms:**
- `ChangeStreamMonitor` receives events from a previously dropped collection
- Resume-after delivers stale or duplicate events

**Cause (fixed in 6.2.2):** The change stream history was not properly purged on collection drop, and in-flight async events could sneak past the drop boundary.

**Solution:** Upgrade to 6.2.2+. The drop operation now advances the sequence counter and purges history both before and after the drop notification.

### Tailable Cursor Returns Empty on PoppyDB

**Symptoms:**
- Tailable cursor `getMore` returns immediately with no data on PoppyDB
- Works fine on real MongoDB

**Cause (fixed in 6.2.2):** `getMore` for tailable cursors did not send `maxTimeMS`, causing PoppyDB to return immediately instead of waiting for new data.

**Solution:** Upgrade to 6.2.2+. The tailable cursor now sends `maxTimeMS=5000` (5s polling interval) by default.

## Getting Help

When reporting issues, include:

1. **Configuration details** (sanitized)
2. **Error logs** with timestamps
3. **Driver statistics** output
4. **MongoDB version** and topology
5. **Java version** and JVM settings
6. **Morphium version**
7. **Steps to reproduce** the issue

```java
// Collect diagnostic information
System.out.println("Morphium version: " + Morphium.getVersion());
System.out.println("Java version: " + System.getProperty("java.version"));
System.out.println("Driver stats: " + morphium.getDriver().getDriverStats());
System.out.println("Configuration: " + morphium.getConfig().toString());
```