Add optional mechanism to exclude files/subdirectories from FileArchive

[sidpalas]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

When using a pulumi FileArchive that references a raw directory, it would be useful to be able to optionally specify a subset of contents of that directory to either include or exclude (via an additional string Attribute on the FileArchive class representing a regex or glob).

The specific use case that inspired this feature request is an AWS lambda function written in python where FileArchive is being used to upload a zip containing the source + requirements.txt files to AWS:

aws.lambda_.Function(
    "api_lambda",
    code=pulumi.FileArchive("../server/"),
    role=iam_for_lambda.arn,
    handler="lambda_function.lambda_handler",
    runtime="python3.9",
)

This works, but the ../server directory also contains a virtual environment that gets included in the resulting .zip file, bloating its size and causing this warning in the AWS interface:

There are some workarounds:

1. Store the virtual environment directory outside of the directory containing the lambda
2. Run the pulumi up command in a CI/CD system that doesn't contain the virtual environment

But it would still be a useful feature.

Affected area/feature

Pulumi SDK (e.g.

pulumi/sdk/python/lib/pulumi/asset.py

Line 99 in 06ba63b

class FileArchive(Archive):

)

[Frassle]

@rawkode suggested also supporting ignore files in #9884.

I'd like to see us support .gitignore, .dockerignore, or arbitrary ignore list of strings within the code itself.

[mrpher]

This would be very useful.

My vote would be for a global .pulumiignore which would be unique for pulumi rather than .gitignore or some existing convention like .gitignore. There could be cases where you'd want to not commit a file to git but still have it visible to pulumi or the other way around.

[MishaKav]

this would be very useful, something like .pulumiignore or even "list of directories or files" to exclude from FileArchive.
In our case, we usually upload a stack from local-machine, so in many directories on MacOS there are file .DS_Store, because of this file, the hash of the directory is changed and pulumi upload many code as new each time.

[cnunciato]

One approach that might help in the meantime would be to use command.local.run with glob expressions. For example, I wanted to upload the source code of a Node app to an S3 bucket, but exclude node_modules, and this seems to work nicely:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as command from "@pulumi/command";

const result = command.local.run({
    command: "echo '🚀'",
    dir: "../my-app",
    archivePaths: [
        "*",
        "!node_modules",
    ],
});

const bucket = new aws.s3.Bucket("bucket");
const runner = new aws.s3.BucketObject("my-app.zip", {
    bucket: bucket.id,
    source: result.then(res => res.archive),
});

[besserwisser]

For this did not work when I wanted to include subdirectories (e.g. a src folder). I also tried something like
"**", "!node_modules"
But it does include the node_modules when using **.