Running spotless without gradle/maven

[fssouza]

Hello @nedtwigg,
While searching for pre commit hook + spotless I came across the following issue:
#178

In there you mentioned:

It wouldn't need to use Gradle, but if it does, this would be a good way to hook it up.

I've tried to find how to do it without using Gradle but I could not figure out. Sorry it might be a silly question but it would help me a lot.

Thanks in advance!

[nedtwigg]

Spotless doesn't have a public static void main() that you can call, but it's portable and easy to embed into a purpose-built main. See How to add a new plugin for a build system for an overview of how to embed Spotless. You can also look at plugin-gradle or plugin-maven to see how it can be integrated.

[fssouza]

Thanks I will check it out

[justintuchek]

This would actually be really nice - especially in a project that has a large number of modules.

Any call to something like ./gradlew spotlessApply will have to eat the entire configuration phase before running the much much shorter spotless task.

In the case where configuration may be 30 seconds to run a 2 second task - that makes having it be accessible as a Gradle plugin less appealing than just a command line file formatter.

[jbduncan]

In the case where configuration may be 30 seconds to run a 2 second task - that makes having it be accessible as a Gradle plugin less appealing than just a command line file formatter.

Also, it occurs to me that it would have the added benefit of allowing Spotless to be more easily integrated into a Bazel rule, which would allow issue #76 to be fulfilled more easily.

Thus I'm inclined to re-open this issue. WDYT @nedtwigg?

[nedtwigg]

If you look at the checklist for adding a new build system, there are two things that stand out as big problems:

• A way to download artifacts from maven, and then run them. Terribly wasteful to do this without an on-disk cache.
• A way to describe a configuration. Name a format, specify which files to format, specify the steps, configure the steps, etc.

Maven and gradle both provide those for free. So do sbt, leiningen, gulp, etc. I don't know much about bazel, but it sure looks like it provides these as built-in utilities.

If you want to build a standalone main, you're going to have to build a good general-purpose implementation of both of those two things, and both are big wide-open problems that require a lot of documentation for users to use. It would be a fun project, but you'll be asking the user to learn your one-off custom Spotless file-format or cli-args for specifying targets and their steps and how to configure each step.

Exploring is always good, but I will be very surprised if there is a large userbase eager to learn this custom one-off tool. Every user you are targeting is already using some kind of build system - the less they have to learn, the easier for them to adopt your work.

Programmers like to design and build big greenfield things (like the Spotless file format mentioned above), but they hate to learn them ;-) If you really want to explore in this direction, I bet that the most promising direction would be to creatively reuse the existing infrastructure. For example, what if every directory had a spotless.gradle file, and you made a Main that reads those using an embedded gradle instance. Now you have a "standalone application", but you don't have to create, document, and teach quite so many things to the user.

[jbduncan]

Wow, you've put a lot of thought into this!

Of course! I'd forgotten that Spotless needs access to Maven Central etc. That indeed mandates that something which can easily download and cache java artifacts e.g. Gradle should be embedded within the "standalone application". But then that would probably add an overhead that would make a Bazel extension using the standalone application not as fast as it reasonably could be.

If I find the time and develop the interest again to work on #76 in the future, I'll research and think about whether it would be possible to port Spotless to Bazel's configuration/extension language Starlark.

[JLLeitschuh]

@jbduncan I think Ktlint has some built in dependency resolution library/tool built in that you could consider using if you really want to go down this route.

But in general, I have a 60+ subproject gradle build and I can tell you pretty confidently (I don't have a build scan to back this up yet) that spotless being configured is not a significant impact on the load time.

In general though, I do agree with @nedtwigg on this one.

[nedtwigg]

spotless being configured is not a significant impact on the load time.

I think that's actually why @jbduncan is considering this change. Spotless is fast, but if it's in a build that is slow to configure, you'll still have to wait for all the slow stuff anyway, unless you're able to somehow split spotless out.

[jbduncan]

I think that's actually why @jbduncan is considering this change. Spotless is fast, but if it's in a build that is slow to configure, you'll still have to wait for all the slow stuff anyway, unless you're able to somehow split spotless out.

Correct, that is more or less what I was thinking. :)

[justintuchek]

Naive question/proposition:

Is it possible to actually point spotless at a different directory outside of the project the task is applied to?

It feels like a bit of a hack but if I could actually make a project that actually just houses spotless to inspect another project (or arbitrary file path) that would be impactful too - it could be a path for users who are already using Gradle and have un-ideal configuration times from their own project.

As a side effect somebody using Bazel could potentially call that from one of their Rules/Task - although it wouldn't be ideal for them to still have a dependency on Gradle - it'd be less of a dependency than having a Gradle based project (I don't wanna pretend to know how Bazel actually works)

[jbduncan]

Hi @jtuchek,

Is it possible to actually point spotless at a different directory outside of the project the task is applied to?

Yes, I think it's possible!

I don't know the exact setup you'd need to create to get this working, but I would be very surprised if it wasn't possible - I believe Gradle is flexible enough to allow this.

However, if running Spotless from a Bazel project (or a non-Gradle/Maven project generally) is what you're after, then I believe I have a slightly better solution for you than running Spotless from outside the project; you could try running Gradle within the project itself, following these very general steps (I may have missed out an important detail or two):

1. create a Gradle wrapper setup somewhere in your project,
2. create a Gradle build file which contains just a Spotless configuration, and put this somewhere in your project,
3. and create a Bazel rule/thingamajig or a script that calls Gradle e.g. path/to/gradlew -b path/to/build.gradle [spotlessCheck|spotlessApply].

I assume that you're reasonably familiar with Gradle, Spotless and the command-line, so if my instructions above don't make sense to you or don't work, I'd be more than happy to clarify things further. (Alternatively, we could continue discussing this via email if you so wish?) :)

Alternatively, if this isn't what you were after, could you explain further what you're trying to achieve exactly?

[jbduncan]

In case this issue is opened again in the future, I think there's an alternative to an embedded Gradle that we could use called Courier, which is a CLI app and Scala API for downloading artifacts from Maven Central.

[perlun]

Bumping into this a few years later, I am one of the ones that would happen to find a standalone CLI useful. But my use case is perhaps a bit special:

I started testing the Google Java Formatter (to reformat some automatically generated code in our Gradle-based code base with a high level of complexity, many subprojects etc). The Google Java Formatter was trivially to integrate in our code generation (standalone Java console app, launched via Gradle):

import com.google.common.collect.Lists;
import com.google.googlejavaformat.java.Main;

public class SomeClass {
    public static void main( String[] args ) throws Exception {
        // Run Java code generation
        //
        // [...]

        List<String> formatterArgs = Lists.newArrayList();
        formatterArgs.add( "--replace" );

        formatterArgs.add( "--aosp" );

        try ( Stream<Path> paths = Files.walk( Paths.get( outputPath ) ) ) {
            paths.filter( Files::isRegularFile )
                    .map( Path::toString )
                    .forEach( formatterArgs::add );
        }

        Main.main( formatterArgs.toArray( new String[0] ) );

Doing the above was fairly trivial since Google Java Formatter provided a nice Main class which I could just invoke with the appropriate arguments. The only problem was that I was not overly happy about the format it uses, which is why I started looking at Spotless as an alternative instead. 🙂

While saying "look at the plugin-gradle as the canonical example" works, it still adds a level of complexity vs just providing something like the above.

Now, I do realize that providing a CLI/console app for something like the Google Java Formatter is a magnitude simpler than it is for Spotless, since the Google tool provides very little configurability. Spotless is an entirely different beast in this regard.

Still, having a dead-simple console application (it could even have a hardwired configuration defined in Java) would be useful for people wanting to integrate the tool like this. Perhaps it's really a niche use case? I don't know, it could be.

[perlun]

Now, I do realize that providing a CLI/console app for something like the Google Java Formatter is a magnitude simpler than it is for Spotless, since the Google tool provides very little configurability. Spotless is an entirely different beast in this regard.

And ah, now (when trying to do this), I realize what you mean @nedtwigg by #268 (comment):

If you look at the checklist for adding a new build system, there are two things that stand out as big problems:

• A way to download artifacts from maven, and then run them. Terribly wasteful to do this without an on-disk cache.

• A way to describe a configuration. Name a format, specify which files to format, specify the steps, configure the steps, etc.

Maven and gradle both provide those for free. So do sbt, leiningen, gulp, etc. I don't know much about bazel, but it sure looks like it provides these as built-in utilities.

So integrating with Spotless from a command-line application (like I'm currently trying to do) is perhaps not so easy... but it can perhaps be done if you just make a no-op Provisioner and load the JARs manually. 🤔 (hmm, no... that doesn't work since a Provisioner which returns an empty Set throws exceptions)

[...]

A while later, I figured out a way that seems to work, in this case hardwired for the Palantir formatter. Posting it here in case it helps others.

public class SomeClass {
    public static void main( String[] args ) throws Exception {
        // Run Java code generation
        //
        // [...]

        List<File> filesToFormat = Lists.newArrayList();
        Path path = Paths.get( outputPath );

        try ( Stream<Path> paths = Files.walk( path ) ) {
            paths.filter( Files::isRegularFile )
                    .map( Path::toFile )
                    .forEach( filesToFormat::add );
        }

        try ( Formatter formatter = Formatter.builder()
                .lineEndingsPolicy( LineEnding.UNIX.createPolicy() )
                .encoding( Charsets.UTF_8 )
                .rootDir( path )
                .steps( ImmutableList.of(
                        createPalantirJavaFormatStep()
                ) )
                .build() ) {

            for ( File file : filesToFormat ) {
                formatter.applyTo( file );
            }
        }
    }

    private static FormatterStep createPalantirJavaFormatStep() {
        return FormatterStep.create(
                "palantir-java-format",
                "",
                state -> new PalantirJavaFormatFormatterFunc()
        );
    }
}

This uses the Palantir default settings. In case you need to tweak it, provide a custom class instead of PalantirJavaFormatFormatterFunc which instantiates Palantir differently. (typical example: different line length than the default of 120 characters) (Noted that this might be a bad example since the line length isn't easily overridable)

[nedtwigg]

@perlun Using Spotless libs to build one-off formatter CLI applications is a great usecase for endusers, and we take our lib API stability seriously to support this exact usecase.

[perlun]

@perlun Using Spotless libs to build one-off formatter CLI applications is a great usecase for endusers, and we take our lib API stability seriously to support this exact usecase.

Sounds good @nedtwigg. 👍 I managed to get the above working (with the Palantir formatting and manually providing the dependencies via our Gradle config), but ended up dismissing the idea since we discussed it internally and had a better idea for that particular problem (reformatting auto-generated code from a 3rd party tool).

---

Straying a bit away from the original topic of this issue, I like what Spotless provides in that it both lets you "test" and "fix" style-related issues though. We are currently using Checkstyle for checking that our Java codebase conforms to a particular set of rules, but all violations have to be fixed manually. In that sense, Spotless would be quite superior in that you can just let it auto-fix the violations.

I guess there isn't any easy "import Checkstyle config" plugin for Spotless or anything? 🙂 Or a "migration guide" of some form?

[jbduncan]

I guess there isn't any easy "import Checkstyle config" plugin for Spotless or anything? 🙂 Or a "migration guide" of some form?

Spotless doesn't, but an external tool called OpenRewrite does!

[perlun]

Thanks @jbduncan, appreciated!