feat(core): add `include` property specifying particular files for assets #33132

go-to-k · 2025-01-24T12:21:25Z

Issue # (if applicable)

Reason for this change

Currently the option for Assets etc. has the exclude only for filtering files. Also I had submitted the PR that adds include before, but closed it because the exclude can cover situations for staging particular files by using negation patterns such as * and !filenames. (see: #26107 (comment)).

But I encountered the situation that it is difficult to express our requirements with exclude only.

The example of directories and files is the following (y: required, x: not required).

- main.py          y
- dummy-1          x
- dummy-2          x
- dummy-3          x
- dummy-...        x
- subdir           
  - examples       x    
    - test.sh      x
    - test.py      x
    - README.md    x
  - a.py           y
  - b.py           y
  - binary         y
  - my.json        y

Now for the situation, we may use the following pattern with exclude only.

exclude: ['*', '!main.py', '!subdir', '!subdir/**/*', 'subdir/examples/*'],

However, there are some confusing points if the double negative is desired:

Negation (!) and positive are mixed and difficult to understand.
It does not work if you change the order in which you specify them (i.e. the double negative no longer works).
- e.g. ['*', 'subdir/examples/*', '!main.py', '!subdir', '!subdir/**/*']
- -> It includes ‘subdir/examples/*’ that should be excluded.

In such cases, the include option makes it simple to achieve the following. (This is very easy to understand and you don't have to mind the order.)

include: ['main.py', 'subdir', 'subdir/**/*'],
exclude: ['subdir/examples/*'],

The include property is also present in other modules that filter particular files, such as cdk watch, codepipeline Git triggers and BucketDeployment for S3, so it isn't unnatural.

In particular, it should be good for developers to be able to match the filtering settings of cdk watch with hotswap and the settings of the asset files added in this PR.

Description of changes

Add an include parameter to interface FileOptions as well as exclude.
Check for matching include in copyDirectory, where the actual file search is done.
If the same file is specified in include and exclude, exclude takes precedence.

Describe any new or updated permissions being added

Description of how you validated changes

Both unit and integ tests.

Checklist

My code adheres to the CONTRIBUTING GUIDE and DESIGN GUIDELINES

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

…include

aws-cdk-automation

(This review is outdated)

comments comments comments rm comments

codecov · 2025-01-24T12:40:56Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.27%. Comparing base (5eeee75) to head (9479e5e).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #33132      +/-   ##
==========================================
+ Coverage   82.20%   82.27%   +0.06%     
==========================================
  Files         119      120       +1     
  Lines        6862     6889      +27     
  Branches     1158     1167       +9     
==========================================
+ Hits         5641     5668      +27     
  Misses       1118     1118              
  Partials      103      103

Flag	Coverage Δ
suite.unit	`82.27% <100.00%> (+0.06%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
packages/aws-cdk	`∅ <ø> (∅)`
packages/aws-cdk-lib/core	`82.27% <100.00%> (+0.06%)`	⬆️

aws-cdk-automation · 2025-02-15T00:11:27Z

This PR has been in the CHANGES REQUESTED state for 3 weeks, and looks abandoned. Note that PRs with failing linting check or builds are not reviewed, please ensure your build is passing

To prevent automatic closure:

Resume work on the PR
OR request an exemption by adding a comment containing 'Exemption Request' with justification e.x "Exemption Request: "
OR request clarification by adding a comment containing 'Clarification Request' with a question e.x "Clarification Request: "

This PR will automatically close in 7 days if no action is taken.

kaizencc · 2025-02-17T17:38:51Z

hi @go-to-k, what's the status on this PR? i don't want to rush you but the automation may eventually close your PR as abandoned...

go-to-k · 2025-02-18T00:32:34Z

hi @kaizencc . Thanks for letting me know!
I've been working on other PRs and put this one off. I'm going to do it later so I'll update the branch so it doesn't get closed.

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

aws-cdk-automation

(This review is outdated)

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

aws-cdk-automation

(This review is outdated)

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

new test

…include

go-to-k · 2025-02-23T13:38:42Z

packages/aws-cdk-lib/core/lib/fs/include.ts

+// eslint-disable-next-line @typescript-eslint/no-require-imports
+const minimatch = require('minimatch');
+
+export function matchIncludePatterns(patterns: string[], absoluteRootPath: string, absoluteFilePath: string): boolean {


Almost the same process as below.
https://github.com/go-to-k/aws-cdk/blob/9479e5ec9327870ec9791eb468c2ae5fdc76d9a5/packages/aws-cdk-lib/core/lib/fs/ignore.ts#L114-L137

go-to-k · 2025-02-23T13:40:03Z

packages/aws-cdk-lib/core/lib/fs/options.ts

+   * If matching the symlink but not its target, it is included (i.e. the `include` patterns match
+   * the symlink path itself, regardless of its target). This is the same behavior even if `follow`
+   * is not `SymlinkFollowMode.NEVER`.


To match the current exclude behavior

aws-cdk-automation · 2025-02-23T13:43:47Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
Commit ID: 9479e5e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

go-to-k · 2025-02-23T14:12:53Z

packages/aws-cdk-lib/core/lib/fs/copy.ts

+      const destDirPath = path.dirname(destFilePath);
+      if (!fs.existsSync(destDirPath)) {
+        fs.mkdirSync(destDirPath, { recursive: true });
+      }


This directory creation code was added because:

Previously, parent directories were already created by the isDirectory() processing before copying files/symlinks (*1).

With the include option to copy only specific files, directories that don't match the include patterns are not created (*2). Without this behavior, we would end up creating unnecessary directories along the traversal path when we finally reach files that aren't included. As a result, we need to explicitly create parent directories for included files/symlinks, particularly to handle cases where those parent directories aren't included.

*1

if (stat && stat.isDirectory()) { fs.mkdirSync(destFilePath); copyDirectory(sourceFilePath, destFilePath, options, rootDir); stat = undefined; }

*2

if (stat && stat.isDirectory()) { if (included) { fs.mkdirSync(destFilePath, { recursive: true }); } copyDirectory(sourceFilePath, destFilePath, options, rootDir); stat = undefined;

go-to-k · 2025-02-23T14:45:23Z

@kaizencc

I've completed the necessary implementation, so I'm ready for review.

go-to-k added 17 commits January 23, 2025 14:57

feat(core): add include property specifying particular files for assets

e54ba69

test name

c1c6f94

Merge branch 'main' of https://github.com/go-to-k/aws-cdk into asset-…

b112e7f

…include

snapshots

15cc85b

snapshots

e0e4ab0

integ

63eca52

README

a571651

docs

e212b48

docs

e149646

change name

73867d0

tweak

c75d5d9

tweak

5a6d36a

new test

3ef2d36

change integ test

6b16e19

change integ

7128e80

README

7e7478b

jsdoc

32b2f0e

github-actions bot added p2 distinguished-contributor labels Jan 24, 2025

aws-cdk-automation requested a review from a team January 24, 2025 12:21

aws-cdk-automation previously requested changes Jan 24, 2025

View reviewed changes

tweak

8eed19e

comments comments comments rm comments

go-to-k force-pushed the asset-include branch from 3719ec5 to 8eed19e Compare January 24, 2025 12:28

Merge branch 'main' into asset-include

3ed34e9

Merge branch 'main' into asset-include

2d040cc

go-to-k added 3 commits February 18, 2025 14:54

Merge branch 'main' into asset-include

94fa585

use require for minimatch match to other files in aws-cdk-lib

fa81af1

style for tests

3c5d386

aws-cdk-automation previously requested changes Feb 18, 2025

View reviewed changes

github-actions bot added effort/medium feature-request labels Feb 18, 2025

go-to-k changed the title ~~feat(core): add include property specifying particular files for assets~~ feat(core): add include property specifying particular files for assets Feb 18, 2025

rm comments in integ

f58d1e3

aws-cdk-automation previously requested changes Feb 18, 2025

View reviewed changes

new test

f07ceb2

new test

go-to-k force-pushed the asset-include branch from 0be48d8 to f07ceb2 Compare February 23, 2025 09:15

go-to-k added 10 commits February 23, 2025 18:21

Merge branch 'main' of https://github.com/go-to-k/aws-cdk into asset-…

370eacb

…include

support nagate with include

e9b8368

error handling

e425044

test

4b9a724

add tests for copy with symlink

3af776c

docs

e5ec40c

refactor tests

adb36d6

add tests

2ba2f87

tests

c7a4cad

docs

9479e5e

go-to-k commented Feb 23, 2025

View reviewed changes

go-to-k marked this pull request as ready for review February 23, 2025 13:45

go-to-k requested a review from a team as a code owner February 23, 2025 13:45

aws-cdk-automation added the pr/needs-community-review label Feb 23, 2025

go-to-k commented Feb 23, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(core): add `include` property specifying particular files for assets #33132

feat(core): add `include` property specifying particular files for assets #33132

go-to-k commented Jan 24, 2025 •

edited

Loading

aws-cdk-automation left a comment •

edited

Loading

codecov bot commented Jan 24, 2025 •

edited

Loading

aws-cdk-automation commented Feb 15, 2025

kaizencc commented Feb 17, 2025

go-to-k commented Feb 18, 2025

aws-cdk-automation left a comment •

edited

Loading

aws-cdk-automation left a comment •

edited

Loading

go-to-k Feb 23, 2025

go-to-k Feb 23, 2025

aws-cdk-automation commented Feb 23, 2025

go-to-k Feb 23, 2025 •

edited

Loading

go-to-k commented Feb 23, 2025 •

edited

Loading

feat(core): add include property specifying particular files for assets #33132

Are you sure you want to change the base?

feat(core): add include property specifying particular files for assets #33132

Conversation

go-to-k commented Jan 24, 2025 • edited Loading

Issue # (if applicable)

Reason for this change

Description of changes

Describe any new or updated permissions being added

Description of how you validated changes

Checklist

aws-cdk-automation left a comment • edited Loading

Choose a reason for hiding this comment

codecov bot commented Jan 24, 2025 • edited Loading

Codecov Report

aws-cdk-automation commented Feb 15, 2025

kaizencc commented Feb 17, 2025

go-to-k commented Feb 18, 2025

aws-cdk-automation left a comment • edited Loading

Choose a reason for hiding this comment

aws-cdk-automation left a comment • edited Loading

Choose a reason for hiding this comment

go-to-k Feb 23, 2025

Choose a reason for hiding this comment

go-to-k Feb 23, 2025

Choose a reason for hiding this comment

aws-cdk-automation commented Feb 23, 2025

AWS CodeBuild CI Report

go-to-k Feb 23, 2025 • edited Loading

Choose a reason for hiding this comment

go-to-k commented Feb 23, 2025 • edited Loading

feat(core): add `include` property specifying particular files for assets #33132

feat(core): add `include` property specifying particular files for assets #33132

go-to-k commented Jan 24, 2025 •

edited

Loading

aws-cdk-automation left a comment •

edited

Loading

codecov bot commented Jan 24, 2025 •

edited

Loading

aws-cdk-automation left a comment •

edited

Loading

aws-cdk-automation left a comment •

edited

Loading

go-to-k Feb 23, 2025 •

edited

Loading

go-to-k commented Feb 23, 2025 •

edited

Loading