[rush] Allow to specify additional files (non tracked by git) for build cache

[lukaskl]

Summary

Previous work:

This ticket is a feature request, which is a follow up of previous conversations:

• codegen + build cache thread on Zulip
• [rush] Support "incrementalBuildAdditionalGlobs" in rush-project.json #3253
• [rush] Specify environment variables for build cache #3730
  • and a PR which closed it: [rush] Configure operation "dependsOnEnvVars" setting #3769

However, none of these tickets above fully encompasses my problem, therefore I'm creating a new ticket.

The problem:

In our company we are using code generation step (Imagine GraphQL or Swagger codegen) where we fetch the schema from the server and generate a client for this server. In our case, we have made a conscious decision to not track neither the schema nor the generated code using git (and we are happy with this choice).

For example, let's imagine 3 projects (A, B and C) - Project C depends on Project B, Project B depends on Project A.
During rush build step:

• Project A - this project fetches the schema from the API and writes it to the file which is gitignored.
• Project B - takes file generated by Project A as an input and generates a server client.
• Project C - is a normal frontend (imagine React, Vue, Angular or similar lib) app which takes generated server client from Project B and generates final bundle.

And in this case, I would love to use build cache feature, just with some tweaks, i.e. I would love to mark Project A as never cacheable, but cache Project B and Project C if the output of project A hasn't changed.

However, this is where I'm encountering the problem - it seems that there is no way to tell rush about additional files (outside of git) which may affect the build cache.
And because Project A is a first project in the chain - basically whole build cache feature is rendered unusable in our case.

Suggestion

Introduce additional dependsOnAdditionalFiles configuration option to operations in rush-project.json.

This option would allow to pass glob (minimatch) patterns pointing to files outside of .git repository. And if this option would be passed, rush would calculate hashes of matched files and include these hashes to the final hash when reading and writing from cache.

I actually have a PR (#3824), which is heavily based another a PR (#3769) introducing a similar feature - dependsOnEnvVars option.

Additional considerations

Undoubtedly, this feature has some pros and cons.

I could see that this feature could be used and abused in various cases, e.g. some may choose to depend on some kind of global file (e.g. /etc/os-release) as a part of the build cache key, some other might choose to build a custom script which generates a custom hash.txt file and then depend on this file. And some of those decisions might end up questionable.

However, at the end of the day, this feature would:

• give a way out from a roadblock
• give a non-opinionated opportunity for repository maintainers to decide themselves how they want to solve build cache problems in case the default options are not enough.

Standard questions

Please answer these questions to help us investigate your issue more quickly:

Question	Answer
@microsoft/rush globally installed version?	5.86.0
rushVersion from rush.json?	5.86.0
useWorkspaces from rush.json?	true
Operating system?	Mac
Would you consider contributing a PR?	Yes
Node.js version (node -v)?	v16.15.0 and v18.12.1