Please document how to serve real credentials in the mocked data

[joshtriplett]

For the purposes of testing an application relying on the IMDS metadata service, I'd like to be able to serve valid AWS credentials via the mocked IMDS.

I'd love to have documentation for how to translate ~/.aws/credentials into the relevant pieces of the mock configuration, or better yet, an option to serve up a particular profile from those credentials as the mocked data.

[slimm609]

@joshtriplett I actually have a blog post about to be published doing exactly this with real credentials and responds the same way as an ec2 instance (all the SDKs pick it up as native instance profile). I am putting the finishing touches on it and should have it done in the next day or so.

Edit: finished it up and published it showing how to do this

https://medium.com/@slimm609/aws-instance-profile-for-local-development-f144b0a7b8b9

[joshtriplett]

@slimm609 That sounds great! Thank you.

[brycahta]

@slimm609 That's a great use case! Would you be open to adding to our Community Use Cases section with the content? Or would you mind if someone from our team added a section if you don't have bandwidth?

In either case, we would attribute you for the content, but I'm reluctant to simply link the blog post because of the overhead for our team to maintain the integrity of said links.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want this issue to never become stale, please ask a maintainer to apply the "stalebot-ignore" label.

[joshtriplett]

Still here, still interested.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want this issue to never become stale, please ask a maintainer to apply the "stalebot-ignore" label.

[github-actions]

This issue was closed because it has become stale with no activity.

[jweyrich]

Speaking on behalf of the company I work for, we're very interested in this too. We use ecs-local-container-endpoints and the aws-sdk-v2 (node) works fine when using the IMDS service from the following projects:

• https://github.com/benkehoe/imds-credential-server
• https://github.com/benkehoe/aws-export-credentials

However, aws-sdk-v3 (node) does not seem to work. The default chain never finds a valid credential from the 2 services above.
We need to inject a real temporary credential for the applications running in our containers to work locally and be debuggable. If we cannot get a real/valid credential, we'd need to mock all services using localstack - not an option currently, since it would require seeding those services with tons of data (s3 with files, dynamodb with tables, and so on).

Our security policies do not allow us to pass AWS_ACCESS_KEY_ID+AWS_SECRET_ACCESS_KEY to our containers.

This is our docker-compose.override.yml:

# Original source from: https://aws.amazon.com/blogs/compute/a-guide-to-locally-testing-containers-with-amazon-ecs-local-endpoints-and-docker-compose/
version: "2.1"
networks:
    default:
        driver: bridge
        ipam:
            driver: default
    credentials_network:
        driver: bridge
        ipam:
            config:
                - subnet: "169.254.170.0/24"
                  gateway: 169.254.170.1
services:
    ecs-local-endpoints:
        image: amazon/amazon-ecs-local-container-endpoints
        volumes:
          - /var/run:/var/run
        env_file:
          - .env
        environment:
          AWS_EC2_METADATA_SERVICE_ENDPOINT: http://host.docker.internal:8081/
          AWS_PROFILE: ${AWS_PROFILE}
        networks:
            credentials_network:
                # This special IP address is recognized by the AWS SDKs and AWS CLI
                ipv4_address: "169.254.170.2"

    app:
        depends_on:
            - ecs-local-endpoints
        networks:
            credentials_network:
                ipv4_address: "169.254.170.3"
        extra_hosts:
          - "host.docker.internal:host-gateway"
        environment:
          # AWS_REGION: ${AWS_REGION} -- We already load using `env_file` in docker-compose.yml
          AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/creds"
          AWS_EC2_METADATA_SERVICE_ENDPOINT: http://host.docker.internal:8081
          ECS_CONTAINER_METADATA_URI: "http://169.254.170.2/v3"

We run either imds-credential-server or aws-export-credentials directly on the host, specifying port 8081 and the AWS profile configured in the host AWS cli - the informed profile uses credential_process along with aws-vault, and that's why we cannot just mount$HOME/.aws as a volume on the container itself and need a service running on the host to retrieve the credential using the credential process, and expose it via IMDS to the containers. We could install aws-vault on the containers just for the debugging session, but we try to keep the debug image as similar as possible to the production image (it's a multi-stage Dockerfile).

As I said, it works fine with sdk v2, but not with v3. Since I haven't figured out the reason yet (already invested 6-8 hours), I went after alternatives and arrived here.

[slimm609]

@jweyrich DId you look at the blog post above? There are several things that are outside the scope of what this tool would do (creating loopback virtual interfaces, templated configs, etc). I created a repo that wraps around ec2-metadata-mock to wrap it all together and works natively on the system with the AWS SDKs. Its in the blog post above as well but here is the repo https://github.com/slimm609/mock-instance-profile

[jweyrich]

@slimm609 I don't think a loopback virtual interface is necessary. See A Guide to Locally Testing Containers with Amazon ECS Local Endpoints and Docker Compose. We define a Docker bridge network in the same CIDR as the IMDS and change the default metadata service endpoint via env var AWS_EC2_METADATA_SERVICE_ENDPOINT in the application container itself. And it appears that the SDK officially uses its value during the credential chain lookup. The new endpoint has to comply with part of IMDS spec - we tested the 2 services I listed above, and they work with the aws-sdk v2, but not with aws-sdk-v3. I'm trying to figure out the reason, or find an IMDS service that would work with both versions of the aws-sdk. Maybe I'm missing a config to make it work on v3 - already tried these providers: fromNodeProviderChain, fromContainerMetadata, fromInstanceMetadata - from @aws-sdk/credential-providers, following what is mentioned in the upgrading docs.

[jweyrich]

@brycahta would it make sense for the project to support providing valid tokens? I ask because the project name contains "mock" - so it may be far from the initial idea/proposal.