Skip to content

[ClangImporter] Respect -working-directory in the created VFS #37946

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 24, 2021

Conversation

bnbarham
Copy link
Contributor

Pass a wrapped VFS down into clang::createInvocationFromCommandLine so
that the working directory is set and then used in the underlying Clang
CompilerInstance.

Fixes the possibility of differing modules hashes when the same
arguments are used in Clang directly vs from the importer.

Resolves rdar://79376364

@bnbarham bnbarham requested review from nkcsgexi and ahoppen June 16, 2021 07:39
@bnbarham
Copy link
Contributor Author

@swift-ci please test

@bnbarham bnbarham requested a review from benlangmuir June 16, 2021 22:27
@benlangmuir
Copy link
Contributor

Is the goal of wrapping the VFS in a RedirectingFileSystem to allow clang to have a different working-directory than Swift? Are we really going to support having two different working directories? I would have expected we would want to be consistent across the whole compiler.

@bnbarham
Copy link
Contributor Author

Is the goal of wrapping the VFS in a RedirectingFileSystem to allow clang to have a different working-directory than Swift? Are we really going to support having two different working directories? I would have expected we would want to be consistent across the whole compiler.

That was the intention, yes.

I started looking back at the previous commits to understand things a bit better. It looks like the way we decided to handle -working-directory was to translate all paths in the driver before passing them to the frontend. Then we pass -Xcc -working-directory to clang for it to handle its relative paths as it sees fit. I believe all that worked fine until recently-ish (2019) where clang now sets the working directory in the underlying FileSystem. Some code paths in FileManager aren't fixing up relative paths (eg. getCanonicalName) and they then use the working directory set in the FileSystem rather than the working directory option (where as they would have previously just used CWD, which would match clang importer's behaviour).

The inconsistency of using the option vs the filesystem should probably be fixed in clang at some point, though it is "correct" since the filesystem also uses the option now (just somewhat fragile/confusing).

On the swift side we could pass -working-directory to the frontend as well and use it to set its FileSystem's CWD (+ using it wherever else workingDirectory is currently passed down from performFrontend). Or have the driver set CWD based on that argument.

There's the also the question of what we should do if the working directories for swift/clang don't match up (ie. -Xcc -working-directory is passed). Should we not allow that now?

Copy link
Contributor

@benlangmuir benlangmuir left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for digging up that context! I think ideally we would pass down -working-directory to the frontend and set it on the VFS, and make it an error to pass a different -Xcc working directory on the CLI. That being said, based on the additional context and looking around more at the implementation, I think your current approach is more conservative and probably less risky if you want to pull it into 5.5. In particular, using the wrapped filesystem will also prevent clang from modifying the CWD of the whole process, which would be bad for SourceKit.

I think before we want to set the working directory across the frontend we need to replace all uses of getRealFileSystem() in our code with createPhysicalFileSystem(), which avoids modifying the process-wide CWD.

@@ -1016,12 +1018,12 @@ ClangImporter::createClangInvocation(ClangImporter *importer,
// to missing files and report the error that clang would throw manually.
// rdar://77516546 is tracking that the clang importer should be more
// resilient and provide a module even if there were building it.
auto VFS = clang::createVFSFromCompilerInvocation(
auto TempVFS = clang::createVFSFromCompilerInvocation(
*CI, *tempClangDiags,
importer->Impl.SwiftContext.SourceMgr.getFileSystem());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we pass in VFS here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes, indeed it should. Thanks, nice catch.

@bnbarham bnbarham force-pushed the importer-use-working-dir branch from 3bc277f to 0be0bb7 Compare June 17, 2021 22:25
@bnbarham bnbarham requested a review from beccadax June 17, 2021 22:30
@bnbarham
Copy link
Contributor Author

Sounds reasonable to me. @nkcsgexi @beccadax did you have any other opinions here?

@bnbarham
Copy link
Contributor Author

@swift-ci please test

@swift-ci
Copy link
Contributor

Build failed
Swift Test OS X Platform
Git Sha - 0be0bb7e6f50fa0580fc81dd5f525982b9c4747c

@swift-ci
Copy link
Contributor

Build failed
Swift Test Linux Platform
Git Sha - 0be0bb7e6f50fa0580fc81dd5f525982b9c4747c

@bnbarham
Copy link
Contributor Author

@swift-ci please test

@bnbarham bnbarham force-pushed the importer-use-working-dir branch from 0be0bb7 to c54bf85 Compare June 22, 2021 23:13
@bnbarham
Copy link
Contributor Author

Ah, only looked at the macOS tests before restarting - the Linux ones failed because of the assert in ClangModuleDependencyScanner.cpp - the temporary createVFSFromCompilerInvocation would end up being a nullptr. I've changed it to use Swift's filesystem in that case instead (since it's just asserting that the arguments are sufficient).

@swift-ci please test

Pass a wrapped VFS down into `clang::createInvocationFromCommandLine` so
that the working directory is set and then used in the underlying Clang
`CompilerInstance`.

Fixes the possibility of differing modules hashes when the same
arguments are used in Clang directly vs from the importer.

Resolves rdar://79376364
@bnbarham bnbarham force-pushed the importer-use-working-dir branch from c54bf85 to 28bb250 Compare June 24, 2021 01:07
@bnbarham
Copy link
Contributor Author

And there were some Windows failures - some / are now \. I wouldn't have expected these changes to alter that at all but they are now correct so... 🤷?

@swift-ci please test

@swift-ci
Copy link
Contributor

Build failed
Swift Test Linux Platform
Git Sha - 28bb250

@bnbarham
Copy link
Contributor Author

Unrelated failure.

@swift-ci please test Linux platform

@swift-ci
Copy link
Contributor

Build failed
Swift Test Linux Platform
Git Sha - 28bb250

@bnbarham
Copy link
Contributor Author

And another...

@swift-ci please test Linux platform

@bnbarham bnbarham merged commit eaf604f into swiftlang:main Jun 24, 2021
@bnbarham bnbarham deleted the importer-use-working-dir branch June 24, 2021 23:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants