[Macros] Add swift-plugin-server executable #64376

rintaro · 2023-03-15T04:17:18Z

(depends on swiftlang/swift-syntax#1411)

Add tools/swift-plugin-server. This executable is intended to be installed in the toolchain and act as an executable compiler plugin just like other 'macro' plugins.

This plugin server has an optional method loadPluginLibrary that dynamically loads dylib plugins.
The compiler has a newly added option -external-plugin-path. This option receives a pair of the plugin library search path (just like -plugin-path) and the corresponding "plugin server" path, separated by #. i.e.

  -external-plugin-path
    <plugin library search path>#<plugin server exectuable path>

For exmaple, when there's a macro decl:

  @freestanding(expression)
  macro stringify<T>(T) -> (T, String) =
      #externalMacro(module: "BasicMacro", type: "StringifyMacro")

The compiler look for libBasicMacro.dylib in -plugin-path paths first, if not found, it falls back to -external-plugin-path and tries to find libBasicMacro.dylib in them. If it's found, the "plugin server" path is launched just like an executable plugin, then loadPluginLibrary method is invoked via IPC, which dlopen the library path in the plugin server. At the actual macro expansion, the mangled name for BasicMacro.SinglifyMacro is used to resolve the macro just like dylib plugins in the compiler.

This is useful for

Isolating the plugin process, so the plugin crashes doesn't result the compiler crash
Being able to use library plugins linked with other swift-syntax versions

rdar://105104850

rintaro · 2023-03-15T06:34:42Z

swiftlang/swift-syntax#1411
@swift-ci Please smoke test

rintaro · 2023-03-15T16:46:05Z

lib/AST/ASTContext.cpp

@@ -67,6 +67,7 @@
 #include "llvm/ADT/Statistic.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringSwitch.h"
+#include "llvm/Config/config.h"


For LTDL_SHLIB_EXT

rintaro · 2023-03-15T16:50:15Z

lib/ASTGen/Sources/ASTGen/PluginHost.swift

+  cxxDiagnosticEngine: UnsafeMutablePointer<UInt8>
+) -> Bool {
+  let plugin =  CompilerPlugin(opaqueHandle: opaqueHandle)
+  assert(plugin.capability.features?.contains("loadPluginLibrary") == true)


This is the only usage of the newly added Capability.features at this point. In case the plugin server doesn't actually implement loadPluginLibrary, the request would just fail.

rintaro · 2023-03-15T21:04:11Z

swiftlang/swift-syntax#1411
@swift-ci Please smoke test

rintaro · 2023-03-15T21:52:33Z

swiftlang/swift-syntax#1411
@swift-ci Please smoke test

rintaro · 2023-03-16T19:08:50Z

swiftlang/swift-syntax#1411
@swift-ci Please smoke test

@freestanding

This executable is intended to be installed in the toolchain and act as an executable compiler plugin just like other 'macro' plugins. This plugin server has an optional method 'loadPluginLibrary' that dynamically loads dylib plugins. The compiler has a newly added option '-external-plugin-path'. This option receives a pair of the plugin library search path (just like '-plugin-path') and the corresponding "plugin server" path, separated by '#'. i.e. -external-plugin-path <plugin library search path>#<plugin server executable path> For exmaple, when there's a macro decl: @freestanding(expression) macro stringify<T>(T) -> (T, String) = #externalMacro(module: "BasicMacro", type: "StringifyMacro") The compiler look for 'libBasicMacro.dylib' in '-plugin-path' paths, if not found, it falls back to '-external-plugin-path' and tries to find 'libBasicMacro.dylib' in them. If it's found, the "plugin server" path is launched just like an executable plugin, then 'loadPluginLibrary' method is invoked via IPC, which 'dlopen' the library path in the plugin server. At the actual macro expansion, the mangled name for 'BasicMacro.StringifyMacro' is used to resolve the macro just like dylib plugins in the compiler. This is useful for * Isolating the plugin process, so the plugin crashes doesn't result the compiler crash * Being able to use library plugins linked with other `swift-syntax` versions rdar://105104850

…est' Separate "load plugin" and "resolve macro" phases. So that the loaded executable plugin is now cached in ASTContext and reused. This saves a 'loadPluginLibrary' IPC messaging when the library provides multiple macros.

Also, build pure swift library/tool with install RPATH like non Swift things.

rintaro · 2023-03-17T17:06:30Z

swiftlang/swift-syntax#1411
@swift-ci Please smoke test

rintaro · 2023-03-17T17:11:40Z

swiftlang/swift-syntax#1411
@swift-ci Please smoke test

bnbarham · 2023-03-17T18:16:13Z

include/swift/AST/ASTContext.h

+  /// Lookup an executable plugin that is declared to handle \p moduleName
+  /// module by '-load-plugin-executable'. Note that the returned path might be
+  /// in the current VFS. i.e. use FS.getRealPath() to get the real path.


Type on equialent. But how about:

Look for dynamic libraries in paths from `-external-plugin-path` and return a pair of `(library path, plugin server executable)` if found. These paths are valid within the VFS, use `FS.getRealPath()` for their underlying path.

One problem with getRealPath here is that it's possible that the overlay has useExternalName set to false, in which case that still won't give the real path 😓.

bnbarham · 2023-03-17T18:50:16Z

include/swift/AST/SearchPathOptions.h

+  /// Paths that contain compiler plugins and the path to the plugin server
+  /// executable.
+  /// e.g. '/path/to/usr/lib/swift/host/plugins#/path/to/usr/bin/plugin-server'.
+  std::vector<std::string> ExternalPluginSearchPaths;


So this std::string is the # separated compiler plugin search path + path to plugin server? If so, maybe:

`#` separated strings containing the compiler plugin search path and path to plugin server.

Or, alternatively, split these when getting the options into either a pair or small custom struct (I'd prefer the latter).

Also, this split seems a little less safe than the path#name one we had previously as it is now two paths rather than a path and a name. IIRC you said that # wasn't valid on all platforms though?

I will make a custom struct.

# is not a special character as a file name at least in macos, (i.e. you can touch ##.txt without errors). I just said it's not common to use # in directory name or filename, and there's no special meaning in shell, unlike ; or ?, etc.
I agree it's not safe, but...

No reason to block on this, # is just a little scary to me because of #1/#2 etc. I can definitely imagine someone doing that 😅

bnbarham · 2023-03-17T18:51:07Z

include/swift/AST/TypeCheckRequests.h

+  }
+
+  void *getAsInProcessPlugin() const {
+    return kind == PluginKind::InProcess ? static_cast<void *>(ptr) : nullptr;


Was the static_cast here just a copy paste? Could use a PointerUnion instead to avoid the need for PluginKind.

Decided not to use PointerUnion because we don't know the alignment of void *

bnbarham · 2023-03-17T19:00:42Z

lib/Driver/ToolChains.cpp

@@ -237,6 +237,7 @@ void ToolChain::addCommonFrontendArgs(const OutputInfo &OI,
  inputArgs.AddAllArgs(arguments, options::OPT_F, options::OPT_Fsystem);
  inputArgs.AddAllArgs(arguments, options::OPT_vfsoverlay);
  inputArgs.AddAllArgs(arguments, options::OPT_plugin_path);
+  inputArgs.AddAllArgs(arguments, options::OPT_external_plugin_path);


Need to implement this in swift-driver too?

Yes we need. Since it's auto generated, I want to do that using main after I merge this PR

bnbarham · 2023-03-17T19:07:46Z

tools/swift-plugin-server/CMakeLists.txt

@@ -0,0 +1,32 @@
+if (SWIFT_SWIFT_PARSER)
+  # _swiftCSwiftPluginServer is just a C support library for  wift-plugin-server


Suggested change

# _swiftCSwiftPluginServer is just a C support library for wift-plugin-server

# _swiftCSwiftPluginServer is just a C support library for swift-plugin-server

rintaro · 2023-03-17T21:22:56Z

@swift-ci Please smoke test

rintaro · 2023-03-17T21:23:35Z

@swift-ci Please smoke test

rintaro · 2023-03-17T21:30:46Z

@swift-ci Please smoke test

* New struct to store a pair of plugin search path and the server path * typo

rintaro · 2023-03-18T02:15:36Z

@swift-ci Please smoke test

rintaro mentioned this pull request Mar 15, 2023

[Plugin] Add 'loadPluginLibrary' message swiftlang/swift-syntax#1411

Merged

rintaro force-pushed the macros-plugin-server branch 4 times, most recently from bceaefe to 1d7148c Compare March 15, 2023 06:28

rintaro marked this pull request as ready for review March 15, 2023 16:12

rintaro requested review from hborla, slavapestov, xedin, artemcm, zoecarver and CodaFi as code owners March 15, 2023 16:12

rintaro requested review from DougGregor and bnbarham and removed request for xedin, slavapestov, CodaFi, artemcm, hborla and zoecarver March 15, 2023 16:12

rintaro commented Mar 15, 2023

View reviewed changes

rintaro force-pushed the macros-plugin-server branch from 6388b96 to 082e0c5 Compare March 15, 2023 21:52

rintaro added 4 commits March 16, 2023 14:00

[Macros] swift-plugin-server cosmetic tweaks

1851ba2

[CMake] Set correct RPATH in add_pure_swift_host_tool

025d874

Also, build pure swift library/tool with install RPATH like non Swift things.

rintaro force-pushed the macros-plugin-server branch from d4869fd to 5b39197 Compare March 17, 2023 17:05

[Macros/Plugin] Make 'features' a set of enum values

54884f0

rintaro force-pushed the macros-plugin-server branch from 5b39197 to 54884f0 Compare March 17, 2023 17:08

bnbarham approved these changes Mar 17, 2023

View reviewed changes

rintaro force-pushed the macros-plugin-server branch from 45db9fb to 9c54eb6 Compare March 17, 2023 21:29

[Macros] Small review changes

b2542a7

* New struct to store a pair of plugin search path and the server path * typo

rintaro force-pushed the macros-plugin-server branch from 9c54eb6 to b2542a7 Compare March 18, 2023 02:15

rintaro merged commit 93e25d2 into swiftlang:main Mar 20, 2023

rintaro mentioned this pull request Apr 26, 2023

[5.9][Macros] Set -external-plugin-path when the toolchain is not Xcode swiftlang/swift-driver#1340

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Macros] Add swift-plugin-server executable #64376

[Macros] Add swift-plugin-server executable #64376

rintaro commented Mar 15, 2023 •

edited

Loading

rintaro commented Mar 15, 2023

rintaro Mar 15, 2023

rintaro Mar 15, 2023

rintaro commented Mar 15, 2023

rintaro commented Mar 15, 2023

rintaro commented Mar 16, 2023

rintaro commented Mar 17, 2023

rintaro commented Mar 17, 2023

bnbarham Mar 17, 2023

bnbarham Mar 17, 2023

rintaro Mar 17, 2023 •

edited

Loading

bnbarham Mar 17, 2023

bnbarham Mar 17, 2023

rintaro Mar 17, 2023

bnbarham Mar 17, 2023

rintaro Mar 17, 2023

bnbarham Mar 17, 2023

rintaro commented Mar 17, 2023

rintaro commented Mar 17, 2023

rintaro commented Mar 17, 2023

rintaro commented Mar 18, 2023

		@@ -0,0 +1,32 @@
		if (SWIFT_SWIFT_PARSER)
		# _swiftCSwiftPluginServer is just a C support library for wift-plugin-server

[Macros] Add swift-plugin-server executable #64376

[Macros] Add swift-plugin-server executable #64376

Conversation

rintaro commented Mar 15, 2023 • edited Loading

rintaro commented Mar 15, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rintaro commented Mar 15, 2023

rintaro commented Mar 15, 2023

rintaro commented Mar 16, 2023

rintaro commented Mar 17, 2023

rintaro commented Mar 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rintaro Mar 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rintaro commented Mar 17, 2023

rintaro commented Mar 17, 2023

rintaro commented Mar 17, 2023

rintaro commented Mar 18, 2023

rintaro commented Mar 15, 2023 •

edited

Loading

rintaro Mar 17, 2023 •

edited

Loading