Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws.RouteTable routes are not notified of changes by aws.Route #2561

Open
rdanno opened this issue Jun 12, 2023 · 8 comments
Open

aws.RouteTable routes are not notified of changes by aws.Route #2561

rdanno opened this issue Jun 12, 2023 · 8 comments
Labels
area/refresh awaiting-upstream The issue cannot be resolved without action in another repository (may be owned by Pulumi). impact/usability Something that impacts users' ability to use the product easily and intuitively kind/bug Some behavior is incorrect or out of spec

Comments

@rdanno
Copy link

rdanno commented Jun 12, 2023

What happened?

Route resources are linked dependencies of RouteTables - but 'inline' Routes don't properly notify the RouteTable when they make a route entry change - and since routes=[] is a property of RouteTable - when you refresh state then the RouteTable sees drift.

I've added ignoreChanges to RouteTable::routes but the refresh still picks up the changes

Expected Behavior

Route should notify RouteTable when it makes changes to its routes so the refresh operation does not need to be run

Steps to reproduce

deploy a routetable with no routes, then use Route to add a route entry then run a refresh

Output of pulumi about

`CLI
Version 3.70.0
Go Version go1.20.5
Go Compiler gc

Plugins
NAME VERSION
aws 5.41.0
python unknown

Host
OS darwin
Version 11.7
Arch x86_64`

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

@rdanno rdanno added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Jun 12, 2023
@danielrbradley
Copy link
Member

Hi @rdanno thanks for writing this up.

If I've understood you correctly ... you're using both inline routes (within the RouteTable resource) and standalone route resources. Then when you do a refresh you see discrepencies because the standalone routes are shown as being removed on the RouteTable. Is that correct.

I believe this is a similar issue to #1790

I think the initial resolution here to to avoid creating routes via the RouteTable property and only use the standalone Route resources. The root cause is likely a limitation of the underlying upstream implementation of this provider.

@danielrbradley danielrbradley added impact/usability Something that impacts users' ability to use the product easily and intuitively and removed needs-triage Needs attention from the triage team labels Jun 13, 2023
@danielrbradley
Copy link
Member

As noted on the terraform resource, this is a known issue in the upstream provider:

Terraform currently provides both a standalone Route resource and a Route Table resource with routes defined in-line. At this time you cannot use a Route Table with in-line routes in conjunction with any Route resources. Doing so will cause a conflict of rule settings and will overwrite rules.

@danielrbradley danielrbradley added the awaiting-upstream The issue cannot be resolved without action in another repository (may be owned by Pulumi). label Jun 13, 2023
@rdanno
Copy link
Author

rdanno commented Jun 13, 2023

Creating the RouteTable with no routes. Adding routes with Route later.

@danielrbradley
Copy link
Member

Ok, that sounds like it's a different issue to the one noted then. We'll need to investigate the upstream implementation to identify if we can address this there.

@t0yv0
Copy link
Member

t0yv0 commented Apr 26, 2024

I have a repro here that I hope demonstrates the issue, unfortunately it is still present in recent Pulumi.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const exampleAwsVpc = new aws.ec2.Vpc("test", {cidrBlock: "10.1.0.0/16"});

const exampleRouteTable = new aws.ec2.RouteTable("example", {
    vpcId: exampleAwsVpc.id,
    // routes: [],
    tags: {
        Name: "example",
    },
});

const egress = new aws.ec2.EgressOnlyInternetGateway("egress", {vpcId: exampleAwsVpc.id});

const exampleRoute = new aws.ec2.Route("route1", {
    routeTableId: exampleRouteTable.id,
    destinationIpv6CidrBlock: "::/0",
    egressOnlyGatewayId: egress.id,
});


export const exampleRouteId = exampleRoute.id;

After the first pulumi up, RouteTable receives inputs nil but outputs as routes: [].

However refreshing this stack shows a warning that RouteTable wants to register the route:

Refresh shows:

Previewing refresh (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/anton-pulumi-corp/aws-2561/dev/previews/d1c94b52-d216-412f-84c4-774e5f61bc72

     Type                                  Name          Plan       Info
     pulumi:pulumi:Stack                   aws-2561-dev
     ├─ aws:ec2:EgressOnlyInternetGateway  egress
     ├─ aws:ec2:Vpc                        test
 ~   ├─ aws:ec2:RouteTable                 example       update     [diff: ~routes]
     └─ aws:ec2:Route                      route1

Resources:
    ~ 1 to update
    4 unchanged

Do you want to perform this refresh?
No resources will be modified as part of this refresh; just your stack's state will be.
 details
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev::aws-2561::pulumi:pulumi:Stack::aws-2561-dev]
    ~ aws:ec2/routeTable:RouteTable: (update)
        [id=rtb-01ff6ea4cd740d0c1]
        [urn=urn:pulumi:dev::aws-2561::aws:ec2/routeTable:RouteTable::example]
        [provider=urn:pulumi:dev::aws-2561::pulumi:providers:aws::default_6_32_0::c96d3f21-e412-40ac-bfd8-c9677165d4fd]
        --outputs:--
      ~ routes         : [
          + [0]: {
                  + carrierGatewayId       : ""
                  + cidrBlock              : ""
                  + coreNetworkArn         : ""
                  + destinationPrefixListId: ""
                  + egressOnlyGatewayId    : "eigw-063e404ff733110ab"
                  + gatewayId              : ""
                  + ipv6CidrBlock          : "::/0"
                  + localGatewayId         : ""
                  + natGatewayId           : ""
                  + networkInterfaceId     : ""
                  + transitGatewayId       : ""
                  + vpcEndpointId          : ""
                  + vpcPeeringConnectionId : ""
                }
        ]

Do you want to perform this refresh?
No resources will be modified as part of this refresh; just your stack's state will be.
  [Use arrows to move, type to filter]
  yes
> no
  details

If the refresh is accepted (our pulumi up --refresh is run), then the system gets into a steady state with clean pulumi preview and refresh, because now the route is duplicated into the outputs of RouteTable as well as into its own resource.

CLI          
Version      3.111.1
Go Version   go1.22.1
Go Compiler  gc

Plugins
NAME    VERSION
aws     6.32.0
awsx    2.9.0
docker  4.5.3
docker  3.6.1
nodejs  unknown

Host     
OS       darwin
Version  14.4.1
Arch     x86_64

This project is written in nodejs: executable='/Users/t0yv0/bin/node' version='v18.18.2'

Current Stack: anton-pulumi-corp/aws-2561/dev

TYPE                                                         URN
pulumi:pulumi:Stack                                          urn:pulumi:dev::aws-2561::pulumi:pulumi:Stack::aws-2561-dev
pulumi:providers:aws                                         urn:pulumi:dev::aws-2561::pulumi:providers:aws::default_6_32_0
aws:ec2/vpc:Vpc                                              urn:pulumi:dev::aws-2561::aws:ec2/vpc:Vpc::test
aws:ec2/egressOnlyInternetGateway:EgressOnlyInternetGateway  urn:pulumi:dev::aws-2561::aws:ec2/egressOnlyInternetGateway:EgressOnlyInternetGateway::egress
aws:ec2/routeTable:RouteTable                                urn:pulumi:dev::aws-2561::aws:ec2/routeTable:RouteTable::example
aws:ec2/route:Route                                          urn:pulumi:dev::aws-2561::aws:ec2/route:Route::route1


Found no pending operations associated with dev

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/anton-pulumi-corp
User           anton-pulumi-corp
Organizations  anton-pulumi-corp, moolumi, pulumi
Token type     personal

Dependencies:
NAME            VERSION
@pulumi/awsx    2.9.0
@pulumi/pulumi  3.113.3
@types/node     18.19.31
typescript      5.4.5
@pulumi/aws     6.32.0

Pulumi locates its logs in /var/folders/gk/cchgxh512m72f_dmkcc3d09h0000gp/T/com.apple.shortcuts.mac-helper// by default

@t0yv0
Copy link
Member

t0yv0 commented Apr 26, 2024

This seems to be very similar to #2246 just for a different pair of coupled resources that are not recommended to be used together, there seems to be a pattern going on here. The issue likely reproduces in TF but is less of a problem there because TF does not warn on refresh of this sort and it refreshes by default when apply is called, so after two terraform apply calls the system gets into a steady state.

@t0yv0
Copy link
Member

t0yv0 commented Apr 26, 2024

Similarly to #2246 ignoreChanges is not able to affect refresh, but there is a new feature being developed for an ignoreRefreshChanges flag that is the recommended workaround for 2246 that should work here as well: pulumi/pulumi#16015

@t0yv0
Copy link
Member

t0yv0 commented Apr 26, 2024

Adding pulumi/pulumi-policy-aws#110 to make the NOTE a visible warning for users of AWSGuard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/refresh awaiting-upstream The issue cannot be resolved without action in another repository (may be owned by Pulumi). impact/usability Something that impacts users' ability to use the product easily and intuitively kind/bug Some behavior is incorrect or out of spec
Projects
None yet
Development

No branches or pull requests

3 participants