Formae Synchronization Issue - Resources Becoming Unmanaged

[CarsonRuns]

Version

v0.75.2

Formae configuration

"@formae/formae.pkl"
import "../vars.pkl"

// Resource imports
import "@aws/ec2/vpc.pkl"


  myvpc = new vpc.VPC {
    label = "vpc-f1b2b189"
    tags = new Listing<formae.Tag> {}
    cidrBlock = "172.31.0.0/16"
    enableDnsHostnames = true
    enableDnsSupport = true
    instanceTenancy = "default"
    target = vars.target.res
    stack = vars.stack.res
  }

Steps to reproduce

See long narrative

Actual behavior

Managed resources become unmanaged

Expected behavior

Managed resources stay managed.

Logs/output

See snippets below.

Additional context

I brought three resources under management and I see them with the inventory command. Then a bit later I run the command again and there is only one resource under management.

AWS::EC2::VPC
AWS::EC2::VPCDHCPOptionsAssociation
AWS::EC2::VPCGatewayAttachment
Only the AWS::EC2::VPCDHCPOptionsAssociation remains under management and both of the other resources show in the list of unmanaged resources. Here is a summary from CoPilot

Formae Synchronization Issue - Resources Becoming Unmanaged

Summary

Resources that were successfully brought under management using formae apply --mode reconcile are being automatically unmarked as managed and moved back to the $unmanaged stack within 2-3 minutes. This occurs during the automatic synchronization process.

Environment

• Formae Version: v0.75.2
• Deployment: Containerized agent with PostgreSQL datastore
• Target: AWS (us-west-2, sandbox profile)
• Stack: sandbox-core
• Affected Resources:
  • VPC: vpc-f1b2b189 (AWS::EC2::VPC)
  • VPC Gateway Attachment: IGW|vpc-f1b2b189 (AWS::EC2::VPCGatewayAttachment)
  • VPC DHCP Options Association: dopt-d3704bab|vpc-f1b2b189 (AWS::EC2::VPCDHCPOptionsAssociation)

Timeline of Events

Initial State (2025-11-24 17:53:29)

Resources discovered and in unmanaged state:

uri: formae://35w2HwH5PipP3tqUvcx95ZzDgID# | vpc-f1b2b189 | managed=f | stack=$unmanaged

Resources Brought Under Management (2025-11-25 17:26:47)

Applied forma file with formae apply --mode reconcile main.pkl:

uri: formae://35w2HwH5PipP3tqUvcx95ZzDgID# | vpc-f1b2b189 | managed=t | stack=sandbox-core

Automatic Unmanagement by Sync (2025-11-25 17:29:19)

Only 2.5 minutes later, synchronization process moved resources back to unmanaged:

uri: formae://35w2HwH5PipP3tqUvcx95ZzDgID# | vpc-f1b2b189 | managed=f | stack=$unmanaged

Database Query Results

Command History

SELECT command_id, command, state, timestamp FROM forma_commands ORDER BY timestamp DESC LIMIT 10;

         command_id          | command |   state    |         timestamp          
-----------------------------+---------+------------+----------------------------
 35yvWR6HiLOsr20GW3ijFT29Fgl | sync    | InProgress | 2025-11-25 18:25:56.632547
 35yuesUGK6qMTND07260voQpSd9 | sync    | Success    | 2025-11-25 18:18:50.915201
 35yub8ECNzVPwIimI3kA68k9 | sync    | Success    | 2025-11-25 18:18:10.497134

Sync commands running approximately every 5-7 minutes

Resource Version History

SELECT uri, label, managed, stack, valid_from FROM resources 
WHERE uri = 'formae://35w2HwH5PipP3tqUvcx95ZzDgID#' ORDER BY valid_from DESC LIMIT 5;

formae://35w2HwH5PipP3tqUvcx95ZzDgID# | vpc-f1b2b189 | f | $unmanaged   | 2025-11-25 17:29:19.238004
formae://35w2HwH5PipP3tqUvcx95ZzDgID# | vpc-f1b2b189 | t | sandbox-core | 2025-11-25 17:26:47.495555
formae://35w2HwH5PipP3tqUvcx95ZzDgID# | vpc-f1b2b189 | f | $unmanaged   | 2025-11-24 17:53:29.04208

Root Cause Analysis

Property Comparison

Compared the resource properties between managed and unmanaged states:

Unmanaged State (from AWS sync):

{
  "VpcId": "vpc-f1b2b189",
  "Tags": [
    {"Key": "FormaeStackLabel", "Value": "sandbox-core"},
    {"Key": "FormaeResourceLabel", "Value": "vpc-f1b2b189"}
  ],
  "CidrBlock": "172.31.0.0/16",
  "InstanceTenancy": "default",
  "EnableDnsSupport": true,
  "EnableDnsHostnames": true
}

Managed State (from forma definition):

{
  "Tags": [
    {"Key": "FormaeStackLabel", "Value": "sandbox-core"},
    {"Key": "FormaeResourceLabel", "Value": "vpc-f1b2b189"}
  ],
  "CidrBlock": "172.31.0.0/16",
  "InstanceTenancy": "default",
  "EnableDnsSupport": true,
  "EnableDnsHostnames": true
}

Key Difference: The unmanaged state includes "VpcId": "vpc-f1b2b189" while the managed state does not.

Forma File Definition

myvpc = new vpc.VPC {
  label = "vpc-f1b2b189"
  tags = new Listing<formae.Tag> {}
  cidrBlock = "172.31.0.0/16"
  enableDnsHostnames = true
  enableDnsSupport = true
  instanceTenancy = "default"
  target = vars.target.res
  stack = vars.stack.res
}

Problem Statement

The synchronization process is detecting drift because:

1. The forma file definition does not include the VpcId property (correctly omitted as it's an AWS-assigned read-only identifier)
2. When sync queries AWS CloudControl API, it receives the resource with the VpcId property included
3. Sync compares the forma definition against the AWS state and treats the presence of VpcId as a difference
4. This triggers the resource to be unmarked as managed and moved back to $unmanaged stack

Configuration

Agent Configuration (formae.conf.pkl):

synchronization {
  enabled = true
  interval = 5.min
}

Expected Behavior

Resources should remain managed after formae apply --mode reconcile unless there are actual configuration drift in user-modifiable properties. Read-only/computed properties like VpcId should be ignored during drift detection.

Actual Behavior

Resources are automatically unmarked as managed every time synchronization runs, making it impossible to maintain managed state for these resources.

Questions for Support

1. Should read-only properties (like VpcId) be ignored during synchronization drift detection?
2. Is it expected behavior to require all AWS-returned properties (including read-only ones) in forma definitions?
3. Is there a configuration option to exclude certain properties from drift detection?
4. Could this be related to the CloudControl API returning additional metadata that shouldn't trigger drift?

Workaround Attempted

None yet - awaiting guidance on whether to:

• Add vpcId property to forma definitions
• Disable synchronization (not ideal for production)
• Use a different mode or configuration setting

Additional Context

• Agent logs show successful startup and PostgreSQL connectivity
• Discovery is working correctly and finding resources
• The issue is reproducible across all three affected resources
• Same behavior occurs after agent restart with fresh container

Impact

Medium - Partial functionality affected

[CarsonRuns]

The container was restarted after the incident, so I don't have access to those logs. Here is more info from the DB.

Formae Agent Logs - Synchronization Unmanagement Incident

Container Restart Log (2025-11-25 17:26:46)

   oooooo      ooooo            oooo     oooo     ooooo       ooo     oooo    
 0oo       o0o0     oo0o      o0o      0oo   ooo0o    0o0   0o   00 0o   00o  
ooo       0oo          o0    0o0      ooo     ooo      oo0  o0    0oo     oo  
ooo      oo0           oo0   oo       oo       oo       oo        o00  o0oo   
oooooo0  ooo            oo   oo       oo       oo       oo    o00 ooo         
ooo       oo           0oo   oo       oo       oo       oo  00    o00    o0o  
ooo        0oo       o0o     oo       oo       oo       oo  o0   oooo0   ooo  
ooo           o00000oo       oo       oo       o0       oo   0000o   0000o      v0.75.2

2025-11-25T17:26:46Z INF Starting agent id=35yoKJ1eMXoaNFBgwOA8YliSDA8
2025-11-25T17:26:46Z INF Started PostgreSQL datastore host=host.docker.internal port=5432 database=formae schema=public user=formae connectionParams=""
2025-11-25T17:26:46Z WRN No secret provided, using random secret, nodes will not be able to communicate secret=16430a7c25021ab8
2025-11-25T17:26:46Z INF Starting actor node node=formae@0.0.0.0
2025-11-25T17:26:46Z INF network is disabled [node=40B61468]
2025-11-25T17:26:46Z INF application 'system_app' (permanent) started [node=40B61468]
2025-11-25T17:26:46Z INF Started PostgreSQL datastore host=host.docker.internal port=5432 database=formae schema=public user=formae connectionParams=""
2025-11-25T17:26:46Z INF Resource synchronization ready [pid=<40B61468.0.1016>] [name='Synchronizer'] [behavior=metastructure.Synchronizer]
2025-11-25T17:26:46Z INF Resource discovery ready [pid=<40B61468.0.1017>] [name='Discovery'] [behavior=discovery.Discovery]
2025-11-25T17:26:46Z INF application 'Application' (permanent) started [node=40B61468]
2025-11-25T17:26:46Z INF node 'formae@0.0.0.0' built with "Ergo Framework:3.1.0" successfully started [node=40B61468]
2025-11-25T17:26:46Z INF Agent started

Database Evidence - Resource Management State Changes

Query: Resource version history for VPC (vpc-f1b2b189)

SELECT uri, label, managed, stack, valid_from 
FROM resources 
WHERE uri = 'formae://35w2HwH5PipP3tqUvcx95ZzDgID#' 
ORDER BY valid_from DESC 
LIMIT 5;

Result:

                  uri                  |    label     | managed |    stack     |         valid_from         
---------------------------------------+--------------+---------+--------------+----------------------------
formae://35w2HwH5PipP3tqUvcx95ZzDgID# | vpc-f1b2b189 | f       | $unmanaged   | 2025-11-25 17:29:19.238004
formae://35w2HwH5PipP3tqUvcx95ZzDgID# | vpc-f1b2b189 | t       | sandbox-core | 2025-11-25 17:26:47.495555
formae://35w2HwH5PipP3tqUvcx95ZzDgID# | vpc-f1b2b189 | f       | $unmanaged   | 2025-11-24 17:53:29.04208

Timeline:

• 17:26:47.495555 - Resource brought under management (managed=true, stack=sandbox-core)
• 17:29:19.238004 - Resource automatically unmanaged (managed=false, stack=$unmanaged)
• Duration: 2 minutes 31 seconds

Query: Synchronization commands during incident window

SELECT command_id, command, state, timestamp 
FROM forma_commands 
WHERE timestamp BETWEEN '2025-11-25 17:25:00' AND '2025-11-25 17:35:00'
ORDER BY timestamp DESC;

Result:

         command_id          | command | state   |         timestamp          
-----------------------------+---------+---------+----------------------------
[No sync commands in logs - sync ran internally without explicit command logging]

Query: All sync commands showing regular 5-minute interval

SELECT command_id, command, state, timestamp 
FROM forma_commands 
ORDER BY timestamp DESC 
LIMIT 10;

Result:

         command_id          | command |   state    |         timestamp          
-----------------------------+---------+------------+----------------------------
 35yvWR6HiLOsr20GW3ijFT29Fgl | sync    | InProgress | 2025-11-25 18:25:56.632547
 35yuesUGK6qMTND07260voQpSd9 | sync    | Success    | 2025-11-25 18:18:50.915201
 35yub8ECNzVPwIimI3kA68k9 | sync    | Success    | 2025-11-25 18:18:10.497134
 35yuZqDSoCMpMcEpFimI3kA68k9 | sync    | Success    | 2025-11-25 18:18:10.497134
 35ytlQAQxBKB376ne0GtimRgbU8 | sync    | Success    | 2025-11-25 18:11:29.708628
 35yt3rSETmhdWK02nETG1JmpdPv | sync    | Success    | 2025-11-25 18:05:42.937265
 35ysw82Dau921utqP7fbmY2FuU6 | sync    | Success    | 2025-11-25 18:04:41.826444
 35ysappYRcRzv5n9c3UnlCuwqdk | sync    | Success    | 2025-11-25 18:01:51.540176
 35yrmbqyTlHBk5U3Yklres35e1B | sync    | Success    | 2025-11-25 17:55:12.616677
 35yqyZxle5h3Ih6xHmVeIqk5oJw | sync    | Success    | 2025-11-25 17:48:33.830286

Pattern: Sync runs approximately every 5-7 minutes

Query: Property comparison between managed and unmanaged states

SELECT data->'Properties' 
FROM resources 
WHERE uri = 'formae://35w2HwH5PipP3tqUvcx95ZzDgID#' 
ORDER BY valid_from DESC 
LIMIT 2;

Result:

Unmanaged State (2025-11-25 17:29:19):

{
  "VpcId": "vpc-f1b2b189",
  "Tags": [
    {"Key": "FormaeStackLabel", "Value": "sandbox-core"}, 
    {"Key": "FormaeResourceLabel", "Value": "vpc-f1b2b189"}
  ], 
  "CidrBlock": "172.31.0.0/16", 
  "InstanceTenancy": "default", 
  "EnableDnsSupport": true, 
  "EnableDnsHostnames": true
}

Managed State (2025-11-25 17:26:47):

{
  "Tags": [
    {"Key": "FormaeStackLabel", "Value": "sandbox-core"}, 
    {"Key": "FormaeResourceLabel", "Value": "vpc-f1b2b189"}
  ], 
  "CidrBlock": "172.31.0.0/16", 
  "InstanceTenancy": "default", 
  "EnableDnsSupport": true, 
  "EnableDnsHostnames": true
}

Key Difference: VpcId property present in unmanaged state but not in managed state

Query: VPC Gateway Attachment state changes

SELECT uri, label, managed, stack, valid_from 
FROM resources 
WHERE uri = 'formae://35w2OjEVZhgIHdegelBQUM66mUs#' 
ORDER BY valid_from DESC 
LIMIT 3;

Result:

                  uri                  |      label       | managed |    stack     |         valid_from         
---------------------------------------+------------------+---------+--------------+----------------------------
formae://35w2OjEVZhgIHdegelBQUM66mUs# | IGW|vpc-f1b2b189 | f       | $unmanaged   | 2025-11-25 17:29:31.211078
formae://35w2OjEVZhgIHdegelBQUM66mUs# | IGW|vpc-f1b2b189 | t       | sandbox-core | 2025-11-25 17:26:46.958751
formae://35w2OjEVZhgIHdegelBQUM66mUs# | IGW|vpc-f1b2b189 | f       | $unmanaged   | 2025-11-24 17:53:34.03128

Same pattern - unmanaged 2 minutes 44 seconds after being brought under management

Query: DHCP Options Association state changes

SELECT uri, label, managed, stack, valid_from 
FROM resources 
WHERE uri = 'formae://35w2MlzNN9PXmPUqFI9A6e7POh4#' 
ORDER BY valid_from DESC 
LIMIT 3;

Result:

                  uri                  |           label            | managed |    stack     |         valid_from         
---------------------------------------+----------------------------+---------+--------------+----------------------------
formae://35w2MlzNN9PXmPUqFI9A6e7POh4# | dopt-d3704bab|vpc-f1b2b189 | t       | sandbox-core | 2025-11-25 17:26:49.008806
formae://35w2MlzNN9PXmPUqFI9A6e7POh4# | dopt-d3704bab|vpc-f1b2b189 | f       | $unmanaged   | 2025-11-24 17:53:25.749823

Note: This resource appears to still be managed as of last query, or was unmanaged in a later sync cycle

Agent Configuration

From formae.conf.pkl:

synchronization {
  enabled = true
  interval = 5.min
}

Discovery Process Logs (showing continued operation)

2025-11-25T17:26:56Z ERR PluginOperator: failed to list resources of type AWS::EC2::NetworkPerformanceMetricSubscription in target sandbox with list parameters map[]: operation error CloudControl: ListResources, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: d08ab4b6-e698-4028-8e50-663875bb6ec0, ThrottlingException: Rate exceeded [pid=<40B61468.0.1045>] [name='formae://discovery-networkperformancemetricsubscription.ec2.aws-35yoLFXI7PjqOVkEl4l1srQaCGI#/List/11cd9ffc-3470-4d7e-8d8f-4d4f613acf62'] [behavior=plugin_operation.PluginOperator]

2025-11-25T17:26:56Z ERR Failed to list resources for AWS::EC2::NetworkPerformanceMetricSubscription in target sandbox: operation error CloudControl: ListResources, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: d08ab4b6-e698-4028-8e50-663875bb6ec0, ThrottlingException: Rate exceeded [pid=<40B61468.0.1017>] [name='Discovery'] [behavior=discovery.Discovery]

2025-11-25T17:40:09Z ERR process terminated abnormally - No handler for message discovery.ResumeScanning in state 'Idle' [pid=<40B61468.0.1017>] [name='Discovery'] [behavior=discovery.Discovery]

2025-11-25T17:40:09Z INF Resource discovery ready [pid=<40B61468.0.2150>] [name='Discovery'] [behavior=discovery.Discovery]

Summary

The database records clearly show:

1. Resources were successfully brought under management at 17:26:47
2. Resources were automatically unmanaged at 17:29:19 (2.5 minutes later)
3. The unmanagement coincides with the synchronization interval (5 minutes)
4. The key difference in resource state is the presence of VpcId property in the AWS-returned state
5. All three resources (VPC, VPCGatewayAttachment, VPCDHCPOptionsAssociation) experienced the same behavior