Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallel parsing - fsc.exe #11140

Merged
merged 13 commits into from
Mar 3, 2021
Merged

Parallel parsing - fsc.exe #11140

Changes from 2 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
d238d25
Enabling parallel parsing for compiling
TIHan Feb 23, 2021
9724f27
Using a delayed error logger per parsing file
TIHan Feb 23, 2021
53f234a
Added -parallel option
TIHan Feb 24, 2021
bf07e86
Fixing error logger
TIHan Feb 24, 2021
4b7c652
Moved parallel compiler option to be a test option
TIHan Feb 25, 2021
d2198df
Trying to get tests to pass
TIHan Feb 25, 2021
d4ad54c
Remove switch
TIHan Feb 25, 2021
23d81e3
Minor refactor
TIHan Feb 25, 2021
3f32f7d
More refactoring
TIHan Feb 25, 2021
03c8b8a
Add comment
TIHan Feb 25, 2021
20f285e
Add max
TIHan Feb 26, 2021
fa5d394
Merged main
TIHan Feb 26, 2021
5c5a466
Handling aggregate exceptions from ArrayParallel. Using try/finally t…
TIHan Mar 2, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 20 additions & 12 deletions src/fsharp/fsc.fs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ open System.IO
open System.Reflection
open System.Text
open System.Threading
open System.Threading.Tasks

open Internal.Utilities
open Internal.Utilities.Filename
Expand Down Expand Up @@ -545,18 +546,25 @@ let main1(ctok, argv, legacyReferenceResolver, bannerAlreadyPrinted,

let inputs =
try
let isLastCompiland, isExe = sourceFiles |> tcConfig.ComputeCanContainEntryPoint

List.zip sourceFiles isLastCompiland
// PERF: consider making this parallel, once uses of global state relevant to parsing are cleaned up
|> List.choose (fun (sourceFile, isLastCompiland) ->

let sourceFileDirectory = Path.GetDirectoryName sourceFile

match ParseOneInputFile(tcConfig, lexResourceManager, ["COMPILED"], sourceFile, (isLastCompiland, isExe), errorLogger, (*retryLocked*)false) with
| Some input -> Some (input, sourceFileDirectory)
| None -> None)

let isLastCompiland, isExe = sourceFiles |> tcConfig.ComputeCanContainEntryPoint
let sourceFiles = isLastCompiland |> List.zip sourceFiles |> Array.ofSeq
let parallelOptions = ParallelOptions(MaxDegreeOfParallelism=min Environment.ProcessorCount sourceFiles.Length)

let results = Array.zeroCreate sourceFiles.Length
Parallel.For(0, sourceFiles.Length, parallelOptions, fun i ->
results.[i] <-
let (filename: string, isLastCompiland) = sourceFiles.[i]
let pathOfMetaCommandSource = Path.GetDirectoryName filename
let delayedErrorLogger = errorLoggerProvider.CreateDelayAndForwardLogger(exiter)
match ParseOneInputFile(tcConfig, lexResourceManager, ["COMPILED"], filename, (isLastCompiland, isExe), delayedErrorLogger, (*retryLocked*)false) with
| Some input -> delayedErrorLogger, Some (input, pathOfMetaCommandSource)
| None -> delayedErrorLogger, None) |> ignore
results
|> Array.choose (fun (delayedErrorLogger, result) ->
cartermp marked this conversation as resolved.
Show resolved Hide resolved
delayedErrorLogger.CommitDelayedDiagnostics errorLogger
result
)
|> List.ofArray
with e ->
errorRecoveryNoRange e
exiter.Exit 1
Expand Down