From 693baa3ff53294f1319409175023aaabe350cad2 Mon Sep 17 00:00:00 2001 From: Daniel Bachler Date: Tue, 23 Oct 2018 09:50:09 +0200 Subject: [PATCH] Allow easy build/format/build cycle of external projects (#233) (#254) * Initial implementation of testing external projects (git clone, build, format, build again). For now tests with the Argu project which fails ATM * Improve building of external projects for regression testing. Add configuration function that allows platform specific configuration of external projects build exectuable and arguments TestExternalProjects is now dependent on Build only for faster testing Platform specific selection of Fanotmas executable from previous build step Shallow cloning of external repos for faster clones; switched to always cloning fresh repo instead of resetting * Fix 452 path * Attempt to add build stages to enable long running external tests on certain conditions in travisCI * Set sudo to true which will AFAIK disable container based builds * Switch back to container based builds, try Argu as an example external project on CI * Reverting sudo to true again since container builds take much more time in startup * Improve build script process name/args handling, fix paths for nix dotnet core testing * Cleanup of accidental changes vs master * More cleanup * Revert accidental whitespace changes in test file * Add testing of external projects only when on a branch named build-external-projects * Final cleanup and comments * Dummy commit to trigger rebuilds after github issues --- .gitignore | 5 +- .travis.yml | 15 ++- README.md | 40 +++--- build.fsx | 124 +++++++++++++++--- src/Fantomas.Cmd/Fantomas.Cmd.fsproj | 2 +- .../Fantomas.CoreGlobalTool.fsproj | 2 +- src/Fantomas.Tests/Fantomas.Tests.fsproj | 2 +- src/Fantomas/Fantomas.fsproj | 2 +- 8 files changed, 144 insertions(+), 48 deletions(-) diff --git a/.gitignore b/.gitignore index 19284ba7c6..fbd7a6a96f 100644 --- a/.gitignore +++ b/.gitignore @@ -177,4 +177,7 @@ paket-files/ # JetBrains Rider .idea/ -*.sln.iml \ No newline at end of file +*.sln.iml + +# external projects directory for extensive regression tests +external-project-tests/ diff --git a/.travis.yml b/.travis.yml index d81c930403..a2723a8aea 100644 --- a/.travis.yml +++ b/.travis.yml @@ -2,11 +2,12 @@ language: csharp dotnet: 2.1.300 mono: latest -sudo: true # use the new container-based Travis infrastructure +sudo: true # as of October 2018 setting this to false is much slower as mono and dotnet are installed into the containers -branches: - only: - - master - -script: - - ./build.sh All +jobs: + include: + - script: + - ./build.sh All + - script: + - ./build.sh TestExternalProjects + if: branch = build-external-projects diff --git a/README.md b/README.md index 3fe31b6696..9b154a4465 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ F# source code formatter, inspired by [scalariform](https://github.com/mdr/scala ## How to use -### Command line tool / API +### Command line tool / API Use this command to install Fantomas as a dotnet SDK global tool: ``` @@ -76,7 +76,7 @@ Fantomas follows the formatting guideline being described in [A comprehensive gu ## Use cases The project is developed with the following use cases in mind: - - Reformatting an unfamiliar code base. It gives readability when you are not the one originally writing the code. + - Reformatting an unfamiliar code base. It gives readability when you are not the one originally writing the code. To illustrate, the following example ```fsharp @@ -93,24 +93,24 @@ To illustrate, the following example will be rewritten to ```fsharp - type Type = + type Type = | TyLam of Type * Type | TyVar of string | TyCon of string * Type list - override this.ToString() = + override this.ToString() = match this with | TyLam(t1, t2) -> sprintf "(%s -> %s)" (t1.ToString()) (t2.ToString()) | TyVar a -> a | TyCon(s, ts) -> s ``` - - Converting from verbose syntax to light syntax. + - Converting from verbose syntax to light syntax. Feeding a source file in verbose mode, Fantomas will format it appropriately in light mode. This might be helpful for code generation since generating verbose source files is much easier. For example, this code fragment ```fsharp - let Multiple9x9 () = + let Multiple9x9 () = for i in 1 .. 9 do printf "\n"; for j in 1 .. 9 do @@ -119,17 +119,17 @@ For example, this code fragment done; done;; Multiple9x9 ();; - ``` - is reformulated to - + ``` + is reformulated to + ```fsharp - let Multiple9x9() = + let Multiple9x9() = for i in 1..9 do printf "\n" for j in 1..9 do let k = i * j printf "%d x %d = %2d " i j k - + Multiple9x9() ``` @@ -138,7 +138,7 @@ For example, this code fragment For more complex examples, please take a look at F# outputs of [20 language shootout programs](tests/languageshootout_output) and [10 CodeReview.SE source files](tests/stackexchange_output). ## Installation -The code base is written in F# 4.X /.NET standard 2.0. +The code base is written in F# 4.X /.NET standard 2.0. The solution file can be opened in Visual Studio 2017, VS Code (with the [ionide plugin](http://ionide.io/)) & [Jetbrains Rider](http://jetbrains.com/rider/). Paket is used to manage external packages. The [test project](src/Fantomas.Tests) depends on FsUnit and NUnit. @@ -146,14 +146,14 @@ However, the [library project](src/Fantomas) and [command line interface](src/Fa ## Testing and validation We have tried to be careful in testing the project. -There are 329 unit tests and 30 validated test examples, +There are 329 unit tests and 30 validated test examples, but it seems some corner cases of the language haven't been covered. Feel free to suggests tests if they haven't been handled correctly. ## Why the name "Fantomas"? -There are a few reasons to choose the name as such. -First, it starts with an "F" just like many other F# projects. -Second, Fantomas is my favourite character in the literature. +There are a few reasons to choose the name as such. +First, it starts with an "F" just like many other F# projects. +Second, Fantomas is my favourite character in the literature. Finally, Fantomas has the same Greek root as "[phantom](https://en.wiktionary.org/wiki/phantom)"; coincidentally F# ASTs and formatting rules are so *mysterious* to be handled correctly. ## How to contribute @@ -168,7 +168,7 @@ Fantomas' features are basically two commands: *format a document* or *format a They both consist in the same two stages: -- parse the code and generate the F# AST (Abstract Syntax Tree). This is provided by the +- parse the code and generate the F# AST (Abstract Syntax Tree). This is provided by the by the FSharp.Compiler.Services library (see the `parse` function in [CodeFormatterImpl.fs](src/Fantomas/CodeFormatterImpl.fs)). - rewrite the code based on the AST (previous step) and formatting settings. @@ -193,7 +193,7 @@ let ``discriminated unions declaration``() = formatSourceString false "type X = private | A of AParameters | B" config |> prepend newline |> should equal """ -type X = +type X = private | A of AParameters | B @@ -222,7 +222,7 @@ that formats the string in input and prints it. The time it took to contribute is sometimes mentioned, as a side note. #### Fixing code generation - - Record type formatting generated invalid F# code in some cases ([#197](https://github.com/dungpa/fantomas/pull/197)) - (2h fix) + - Record type formatting generated invalid F# code in some cases ([#197](https://github.com/dungpa/fantomas/pull/197)) - (2h fix) ## Credits We would like to gratefully thank the following persons for their contributions. @@ -237,5 +237,5 @@ We would like to gratefully thank the following persons for their contributions. - [Enrico Sada](https://github.com/enricosada) ## License -The library and tool are available under Apache 2.0 license. +The library and tool are available under Apache 2.0 license. For more information see the [License file](LICENSE.md). diff --git a/build.fsx b/build.fsx index c3d7608088..3d4ab158e1 100644 --- a/build.fsx +++ b/build.fsx @@ -1,5 +1,5 @@ // -------------------------------------------------------------------------------------- -// FAKE build script +// FAKE build script // -------------------------------------------------------------------------------------- #r @"packages/build/FAKE/tools/FakeLib.dll" @@ -10,12 +10,12 @@ open System // Git configuration (used for publishing documentation in gh-pages branch) -// The profile where the project is posted +// The profile where the project is posted let gitHome = "https://github.com/dungpa" // The name of the project on GitHub let gitName = "fantomas" -// The name of the project +// The name of the project // (used by attributes in AssemblyInfo, name of a NuGet package and directory in 'src') let project = "Fantomas" @@ -32,11 +32,11 @@ let configuration = "Release" // Longer description of the project // (used as a description for NuGet package; line breaks are automatically cleaned up) -let description = """This library aims at formatting F# source files based on a given configuration. -Fantomas will ensure correct indentation and consistent spacing between elements in the source files. -Some common use cases include -(1) Reformatting a code base to conform a universal page width -(2) Converting legacy code from verbose syntax to light syntax +let description = """This library aims at formatting F# source files based on a given configuration. +Fantomas will ensure correct indentation and consistent spacing between elements in the source files. +Some common use cases include +(1) Reformatting a code base to conform a universal page width +(2) Converting legacy code from verbose syntax to light syntax (3) Formatting auto-generated F# signatures.""" // List of author names (for NuGet package) @@ -50,12 +50,47 @@ let solutionFile = "fantomas" // Environment.CurrentDirectory <- __SOURCE_DIRECTORY__ let release = parseReleaseNotes (IO.File.ReadAllLines "RELEASE_NOTES.md") +// Types and helper functions for building external projects (see the TestExternalProjects target below) +type ProcessStartInfo = + { ProcessName : string + Arguments : string list } + +type ExternalProjectInfo = + { GitUrl : string + DirectoryName : string + Tag : string + SourceSubDirectory : string + BuildConfigurationFn : (string -> ProcessStartInfo) } + +// Construct the commands/arguments for running an external project build script for both windows and linux +// For linux we run this by invoking sh explicitly and passing the build.sh script as an argument as some +// projects generated on windows don't have the executable permission set for .sh scripts. On windows we +// treat .cmd files as executable +let configureBuildCommandFromDefaultFakeBuildScripts pathToProject = + if Fake.EnvironmentHelper.isWindows + then { ProcessName = pathToProject "build.cmd"; Arguments = [ "Build" ] } + else { ProcessName = "sh"; Arguments = [ sprintf "%s/build.sh Build" pathToProject ] } + +// Construct the path of the fantomas executable to use for external project tests +let fantomasExecutableForExternalTests projectdir = + if Fake.EnvironmentHelper.isWindows + then { ProcessName = sprintf "%s/src/Fantomas.Cmd/bin/%s/net452/dotnet-fantomas.exe" projectdir configuration; Arguments = [] } + else { ProcessName = "dotnet"; Arguments = [ sprintf "%s/src/Fantomas.CoreGlobalTool/bin/%s/netcoreapp2.1/fantomas-tool.dll" projectdir configuration ] } + +let externalProjectsToTest = [ + { GitUrl = @"https://github.com/fsprojects/Argu" + DirectoryName = "Argu" + Tag = "5.1.0" + SourceSubDirectory = "src" + BuildConfigurationFn = configureBuildCommandFromDefaultFakeBuildScripts } + ] + // -------------------------------------------------------------------------------------- // Clean build results & restore NuGet packages Target "Clean" (fun _ -> CleanDirs [ - "bin" + "bin" "nuget" "src/Fantomas/bin" "src/Fantomas/obj" @@ -71,7 +106,7 @@ Target "AssemblyInfo" (fun _ -> [ Attribute.Product project Attribute.Description summary Attribute.Version release.AssemblyVersion - Attribute.FileVersion release.AssemblyVersion ] + Attribute.FileVersion release.AssemblyVersion ] CreateFSharpAssemblyInfo "src/Fantomas/AssemblyInfo.fs" ( Attribute.InternalsVisibleTo "Fantomas.Tests" :: Attribute.Title "FantomasLib" :: shared) @@ -94,8 +129,8 @@ Target "ProjectVersion" (fun _ -> // Build library & test project Target "Build" (fun _ -> DotNetCli.Build (fun p -> - { p with - Project = (sprintf "src/%s.sln" solutionFile) + { p with + Project = (sprintf "src/%s.sln" solutionFile) Configuration = configuration } ) @@ -103,12 +138,12 @@ Target "Build" (fun _ -> Target "UnitTests" (fun _ -> DotNetCli.Test (fun p -> - { p with + { p with Project = "src/Fantomas.Tests/Fantomas.Tests.fsproj" Configuration = configuration AdditionalArgs = ["--no-build --no-restore --test-adapter-path:. --logger:nunit;LogFilePath=../../TestResults.xml"] } - ) + ) ) // -------------------------------------------------------------------------------------- @@ -136,12 +171,66 @@ Target "Pack" (fun _ -> ] |> String.concat " " "pack src/"+project+"/"+project+".fsproj -c "+ configuration + " -o ../../bin " + packParameters |> DotNetCli.RunCommand id - + pack "Fantomas" pack "Fantomas.Cmd" pack "Fantomas.CoreGlobalTool" ) + +// This takes the list of external projects defined above, does a git checkout of the specified repo and tag, +// tries to build the project, then reformats with fantomas and tries to build the project again. If this fails +// then there was a regression in fantomas that mangles the source code + +Target "TestExternalProjects" (fun _ -> + let externalBuildErrors = + externalProjectsToTest + |> List.map (fun project -> + let relativeProjectDir = sprintf "external-project-tests/%s" project.DirectoryName + + Fake.FileHelper.CleanDir relativeProjectDir + // Use "shallow" clone by setting depth to 1 to only check out the one commit we want to build + Fake.Git.CommandHelper.gitCommand "." (sprintf "clone --branch %s --depth 1 %s %s" project.Tag project.GitUrl relativeProjectDir) + + let fullProjectPath = sprintf "%s/%s" __SOURCE_DIRECTORY__ relativeProjectDir + let buildStartInfo = project.BuildConfigurationFn fullProjectPath + + let buildExternalProject() = ExecProcess + (fun info -> info.FileName <- buildStartInfo.ProcessName + info.WorkingDirectory <- relativeProjectDir + info.Arguments <- String.Join(" ", buildStartInfo.Arguments)) + (TimeSpan.FromMinutes 5.0) + + let cleanResult = buildExternalProject() + if cleanResult <> 0 then failwithf "Initial build of external project %s returned with a non-zero exit code" project.DirectoryName + + let fantomasStartInfo = + fantomasExecutableForExternalTests __SOURCE_DIRECTORY__ + let arguments = + fantomasStartInfo.Arguments @ [ sprintf "--recurse %s" project.SourceSubDirectory ] + |> fun args -> String.Join(" ", args) + let invokeFantomas() = ExecProcess + (fun info -> info.FileName <- fantomasStartInfo.ProcessName + info.WorkingDirectory <- sprintf "%s/%s" __SOURCE_DIRECTORY__ relativeProjectDir + info.Arguments <- arguments) + (TimeSpan.FromMinutes 5.0) + let fantomasResult = invokeFantomas() + + if fantomasResult <> 0 + then Some <| sprintf "Fantomas invokation for %s returned with a non-zero exit code" project.DirectoryName + else + let formattedResult = buildExternalProject() + if formattedResult <> 0 + then Some <| sprintf "Build of external project after fantomas formatting failed for project %s" project.DirectoryName + else + printfn "Successfully built %s after reformatting" project.DirectoryName + None + ) + |> List.choose id + if not (List.isEmpty externalBuildErrors) + then failwith (String.Join("\n", externalBuildErrors) ) +) + Target "Push" (fun _ -> Paket.Push (fun p -> { p with WorkingDir = "bin" })) // -------------------------------------------------------------------------------------- @@ -158,4 +247,7 @@ Target "All" DoNothing ==> "All" ==> "Push" -RunTargetOrDefault "All" \ No newline at end of file +"Build" + ==> "TestExternalProjects" + +RunTargetOrDefault "All" diff --git a/src/Fantomas.Cmd/Fantomas.Cmd.fsproj b/src/Fantomas.Cmd/Fantomas.Cmd.fsproj index 367c891b7e..3ec99bc8e7 100644 --- a/src/Fantomas.Cmd/Fantomas.Cmd.fsproj +++ b/src/Fantomas.Cmd/Fantomas.Cmd.fsproj @@ -17,4 +17,4 @@ - \ No newline at end of file + diff --git a/src/Fantomas.CoreGlobalTool/Fantomas.CoreGlobalTool.fsproj b/src/Fantomas.CoreGlobalTool/Fantomas.CoreGlobalTool.fsproj index 0a37642eb8..f493a0ff0d 100644 --- a/src/Fantomas.CoreGlobalTool/Fantomas.CoreGlobalTool.fsproj +++ b/src/Fantomas.CoreGlobalTool/Fantomas.CoreGlobalTool.fsproj @@ -18,4 +18,4 @@ - \ No newline at end of file + diff --git a/src/Fantomas.Tests/Fantomas.Tests.fsproj b/src/Fantomas.Tests/Fantomas.Tests.fsproj index 65812180f9..026ba6c89b 100644 --- a/src/Fantomas.Tests/Fantomas.Tests.fsproj +++ b/src/Fantomas.Tests/Fantomas.Tests.fsproj @@ -46,4 +46,4 @@ - \ No newline at end of file + diff --git a/src/Fantomas/Fantomas.fsproj b/src/Fantomas/Fantomas.fsproj index 8910bd6b59..9facfd66b5 100644 --- a/src/Fantomas/Fantomas.fsproj +++ b/src/Fantomas/Fantomas.fsproj @@ -21,4 +21,4 @@ - \ No newline at end of file +