Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor uses & creation of temp files/dirs to use unified API #15240

Merged
merged 1 commit into from Dec 9, 2020

Conversation

JLLeitschuh
Copy link
Contributor

@JLLeitschuh JLLeitschuh commented Nov 19, 2020

@JLLeitschuh JLLeitschuh requested a review from a team as a code owner November 19, 2020 20:49
@JLLeitschuh JLLeitschuh requested a review from a team November 19, 2020 20:50
@gradle gradle deleted a comment from JLLeitschuh Nov 19, 2020
@gradle gradle deleted a comment from JLLeitschuh Nov 19, 2020
@JLLeitschuh JLLeitschuh force-pushed the improve/JLL/temp_dir_cleanup branch 2 times, most recently from 68ae1ea to 7797fc0 Compare November 19, 2020 23:42
@gradle gradle deleted a comment from JLLeitschuh Nov 19, 2020
@gradle gradle deleted a comment from JLLeitschuh Nov 20, 2020
@gradle gradle deleted a comment from JLLeitschuh Nov 20, 2020
@gradle gradle deleted a comment from JLLeitschuh Nov 21, 2020
Copy link
Member

@lptr lptr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I started reviewing this and left a bunch of comments, but the single change here is way too big, and it's hard to tell what belongs where. It would be great if you could split this up into several smaller commits.

.teamcity/subprojects.json Outdated Show resolved Hide resolved
implementation(project(":base-services"))

implementation(libs.inject)
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think working with temporary files requires a separate subproject. Indeed, we have the :files subproject that already contains classes in the org.gradle.internal.file package. I think these should go there, perhaps under org.gradle.internal.file.temp.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code needed to make its way into a variety of places, like workers and such, and I didn't want to be adding more dependencies than were strictly necessary when doing it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think :files is already a dependency of everything where you'd need this. What does not depend on it that still needs temporary files?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The projects jvm-services & native don't depend upon the entirety of files and I didn't want to create a coupling here on more than was necessary. My goal here was to apply the single-responsibility principle so that downstream consumers are coupled to the fewest dependencies. Thus allowing Gradle to continue to be able to parallelize the compilation of these components. If I were to introduce this explicit coupling, compilation wouldn't be able to happen in parallel.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very much commendable, however with our current setup adding a new JAR adds some non-trivial costs, too, both at build time and at runtime. The most significant of these is the lengthening of the Windows classpath, which causes innumerable pains. We should fix this for good at some point by combining our JARs into a single fat one or something, but we don't seem to be getting around to doing this.

In any case I don't think parallel compilation for these three classes is a major issue that would warrant a separate subproject by itself.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran this by @big-guy and he gave me guidance to continue with the separate project for now. You're not wrong about the issues here with the the windows paths. We've decided we can always re-combine these if we have issues moving forward.

});
}

private TmpDirTemporaryFileProvider(final Factory<File> tempDirProvider) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We now use Tmp (in TmpDirTemporaryFileProvider), Temp (in TempFiles) and Temporary to name these types. This line has all three variations.

I think it would be great to standardize on one way instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hopefully this won't last much longer.

import java.io.File;

@ServiceScope(Scopes.UserHome.class)
public class TmpDirTemporaryFileProvider extends DefaultTemporaryFileProvider {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could call this a DirectoryBasedTemporaryFileProvider to avoid repeating "temporary".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is this different to DefaultTemporaryFileProvider?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DefaultTemporaryFileProvider is for the underlying logic for creating the temporary files. The TmpDirTemporaryFileProvider is an implementation detail that the files are actually coming from the system temporary directory. After this PR is merged I'm going to be working on trying to get rid of the actual uses of this class.

TmpDir - It's coming from the system temp directory.
TemporaryFileProvider - It's implementing the TemporaryFileProvider interface.

The plan is to move Gradle to create and use a temporary directory under the Gradle User Home moving forward.

subprojects/core/build.gradle.kts Outdated Show resolved Hide resolved
@JLLeitschuh JLLeitschuh force-pushed the improve/JLL/temp_dir_cleanup branch 4 times, most recently from 66ca5d1 to 5f24f7c Compare December 3, 2020 14:07
@gradle gradle deleted a comment from JLLeitschuh Dec 3, 2020
@gradle gradle deleted a comment from JLLeitschuh Dec 3, 2020
@gradle gradle deleted a comment from JLLeitschuh Dec 3, 2020
@gradle gradle deleted a comment from JLLeitschuh Dec 3, 2020
@JLLeitschuh JLLeitschuh force-pushed the improve/JLL/temp_dir_cleanup branch 2 times, most recently from f28bc4a to 22ec917 Compare December 4, 2020 18:13
@gradle gradle deleted a comment from JLLeitschuh Dec 4, 2020
@gradle gradle deleted a comment from JLLeitschuh Dec 7, 2020
@gradle gradle deleted a comment from JLLeitschuh Dec 7, 2020
@JLLeitschuh JLLeitschuh changed the base branch from release to master December 7, 2020 21:13
@gradle gradle deleted a comment from JLLeitschuh Dec 7, 2020
@JLLeitschuh JLLeitschuh force-pushed the improve/JLL/temp_dir_cleanup branch 2 times, most recently from 2460e76 to 746b6d5 Compare December 8, 2020 16:40
This centralization allows unified control over temporary file & directory creation.
The goal is to drive all temp directory creation logic through a
set of common logical components. Doing so allows us to make our temp directory
usage relocatable.
@gradle gradle deleted a comment from JLLeitschuh Dec 8, 2020
@gradle gradle deleted a comment from JLLeitschuh Dec 8, 2020
@gradle gradle deleted a comment from JLLeitschuh Dec 8, 2020
@JLLeitschuh
Copy link
Contributor Author

@bot-gradle test QuickFeedback

@bot-gradle
Copy link
Collaborator

OK, I've already triggered QuickFeedback build for you.

@JLLeitschuh
Copy link
Contributor Author

@bot-gradle test this please

@bot-gradle
Copy link
Collaborator

OK, I've already triggered ReadyForMerge build for you.

@big-guy big-guy merged commit c0f43f2 into gradle:master Dec 9, 2020
14 checks passed
@JLLeitschuh JLLeitschuh added this to the 7.0 RC1 milestone Dec 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants