diff --git a/Docs/clustering.adoc b/Docs/clustering.adoc index 55d6b0ed7..adfda45a9 100644 --- a/Docs/clustering.adoc +++ b/Docs/clustering.adoc @@ -1,4 +1,5 @@ +[[clustering]] == Clustering > One actor is no actor, they come in systems. @@ -6,6 +7,7 @@ By connecting several nodes into a cluster you gain the ability to _transparently_ send and receive messages from actors located on other nodes. Along with this, advanced failure detection and mitigation mechanisms are also enabled by default. +[[cluster_quickstart]] === Cluster QuickStart ==== Step 1: Start nodes and join them diff --git a/Docs/serialization.adoc b/Docs/serialization.adoc index 1aef2414c..5330e8d29 100644 --- a/Docs/serialization.adoc +++ b/Docs/serialization.adoc @@ -1,16 +1,87 @@ [[serialization]] == Serialization -> Simple serialization layer which decouples business logic from low level serialization concerns. -> Built-in support for swift:Codable[], https://developers.google.com/protocol-buffers/[Protocol Buffers], as well as custom serializers. +> The Actor system provides _transparent serialization_, allowing to avoid entangling of business logic and any serialization (coding, decoding) logic with your actors. -In order to go distributed with actors you will have to give at least a little bit of thought (and for clustered -production systems, actually quite a lot of thought!) on how the messages sent between actors should be serialized. +Serialization of actor messages is handled transparently, meaning that you can simply send https://developer.apple.com/documentation/foundation/archives_and_serialization/encoding_and_decoding_custom_types[Codable] messages +to other actors and they will decode and receive them properly, without any additional coding work needing to be done. -Swift Distributed Actors offers built-in support for https://developer.apple.com/documentation/foundation/archives_and_serialization/encoding_and_decoding_custom_types[Codable] -messages and serialization using https://developers.google.com/protocol-buffers/[Protocol Buffers]. However, it does -not restrict or require the use of these. If you want to use a different serialization mechanism for all messages, -or just a few specific messages, it is entirely possible thanks to the serializer registration mechanism. +Serialization is only employed to messages which are sent across process or network boundaries (unless `serializeLocalMessages` is enabled). Helper types are provided for encoding messages using <>, and any custom serializer (e.g. flat buffers, or your custom binary format) are possible to plug-in on a type-by-type basis. + +NOTE: For security reasons, in **production environments** we strongly recommend registering all types which are meant for serialization, for details on how to do this see <>. + +[[quickstart_serialization]] +=== Quickstart: Sending (Remote) Messages + +In order to take your local-only actor system, to the next level and allow its actors to communicate over the network, +you should follow the following 3 steps: + +- Step 1: Ensure that your system has multiple nodes to host your actors as explained in <>, +- Step 2: Since all actor messages are enforced to be `Codable`, you can send them _just the same way as before_ to (now remote) actors: `remoteActor.hello()` (or when using ActorRefs: `remoteActorRef.tell(.hello)`), + * In order to locate actors on different nodes, you can use the <>, +- Step 3: "There's no step 3." https://www.youtube.com/watch?v=YHzM4avGrKI[*] + +Having that said, there are a few more steps to take when deploying an actor system into **production**, which we'll explore in the following section. + +[[serialization_production]] +=== Serialization considerations for Production Deployments + +By default in DEBUG mode, the system operates in `inboundSerializerManifestMappings` mode (see api:Serialization.Settings[struct]), +which will log warnings whenever a "not registered" type is sent or received; the system will however to attempt to (de-)serialize +any such message. + +In release mode (when building with `swift build -c release`), the `inboundSerializerManifestMappings` is automatically changed to +`false`, meaning that messages which have not been registered fail to (de-)serialize. + +In order to register messages for a production deployment, it is recommended to work locally in debug mode, gather logs which highlight which messages are needed to be sent, and then codify this in the system's setup, like shown below: + +[source] +---- +include::{dir_sact_doc_tests}/SerializationDocExamples.swift[tag=serialization_register_types] +---- + +#TODO: We want to enable registering an entire module, see: https://github.com/apple/swift-distributed-actors/issues/547[Serialization: allow to "trust" entire module #547]# + + +=== Codable Messages + +As messages are constrained to conform to `Codable`, all messages are in principle able to be sent over the wire. +Serialization however is only applied when messages cross process or network boundaries. + +NOTE: While messages locally are passed simply by storing the passed message in the recipients mailbox and thus technically _could_ store references to shared state -- beware of such designs as they lead to hard to make distributed (or process isolated) designs, and will break the actor's concurrency guarantees if the shared data-structure is mutable. + +==== Selecting the default Codable Coders + +The ActorSystem's serialization infrastructure allows for specifying what coder should be used for certain messages. + +If a type is safe to be serialized (see <>), it will be serialized using the serializer associated with its type. If no serializer is specified for its specific type, the default Codable coder will be used. + +You can configure which coder should be used by default when encoding messages by setting `settings.serialization.defaultSerializer`. + +It is possible to specify a different serializer for specific messages, this is done by: + +[source] +---- +include::{dir_sact_doc_tests}/SerializationDocExamples.swift[tag=serialization_specific_coder] +---- + + +==== Manually implementing Codable for Enums with Associated Values + +WARNING: Messages are most often enums with associated values, for which Swift does not synthesize Codable conformances today. We are aiming to solve this in Swift itself by improving its synthesising capabilities, please bear with us while we work to make this happen. + +Most of the time you should be able to lean on Swift's synthesis of Codable conformances. However, sometimes you may have +to implement such conformance yourself. While there is plenty guides about doing so online, we would like to explicitly +outline the suggested style of encoding enums with associated values (as many messages often are). + +NOTE: When using <> you do not need ot implement the conformance for `MyActorable.Message` because it is generated for you using the `GenActors` source generator. + +We suggest the use of the following pattern to encoding enums with associated values: + +[source] +---- +include::{dir_sact_doc_tests}/SerializationDocExamples.swift[tag=serialization_codable_manual_enum_assoc] +---- === Which Serializer to Pick? @@ -93,14 +164,8 @@ include::{dir_sact_doc_tests}/SerializationDocExamples.swift[tag=prepare_system_ The `settings` argument passed to the configuration closure is an api:ActorSystemSettings[struct], which can be used to configure various parts of the actor system. -NOTE: While it would be nice to avoid this registration step, it does not seem possible, as `Codable` infrastructure - requires the type to be present for invoking the decoding phase. In order to be able to get the type back when - deserializing an incoming message, this mapping is used. + - + - #note: On the other hand, automatic ID assigning is inherently not entirely safe nor a good idea for building a production ready system, -so in some ways, it is good that we teach immediately about this need. Needs more docs though# - -NOTE: Serialization identifiers until 1000 are reserved for Swift Distributed Actors's internal use. #TODO sadly we have to register for every specific type... This is the correct thing to do for protocol evolution, but makes getting started harder. We can't register for "any codable" (or I can't figure out how), due to how Swift's MetaTypes work.# +NOTE: Serialization identifiers until 16 are reserved for Swift Distributed Actors's internal use. + If you want to register a different serialization format, please use numbers from greater or equal to 17. ==== Step 4: Send messages as usual @@ -126,79 +191,6 @@ WARNING: Due to this permissive style of sending messages, it technically is pos + It is possible to configure `Serialization` to always serialize all messages, so you would catch such mistakes even when testing locally. <> -[[serialization_codable]] -=== Serializing Codable Messages - -The swift:Codable[] protocol can be used for automatic (although customizable) derivation of serializers for Swift data types. -Refer to its reference documentation in https://developer.apple.com/documentation/foundation/archives_and_serialization/encoding_and_decoding_custom_types[Encoding and Decoding Custom Types] -to learn more about how to use the protocol and its extension points, as this guide will assume prior knowledge of how `Codable` works. - -==== Step 1: Conform messages to `Codable` - -Suppose we have a `driver` actor and its message protocol consists of a single `ParkingSpotStatus` enum: - -.Example: Conform message to Codable -[source] ----- -include::{dir_sact_doc_tests}/SerializationDocExamples.swift[tag=serialization_codable_messages] ----- - -Note that we immediately conform it to swift:Codable[]. Since the type is a simple enum without associated values -the serialization code for it is generated automatically. - -==== Step 2: Register `Codable` types for serialization - -In order to let Swift Distributed Actors know which serialization engine to use, a serializer has to be registered for each type. -Since it is very important that the _right_ serializer is picked up by another node in a cluster, serializers _must_ be -given unique identifiers (which matters in case of upgrading your system where the message types change). - -To register the message types intended for use in the cluster they have to be registered at system bootstrap, -using the optional trailing settings configuration closure: - -.Configuring ActorSystem to use Codable serializers for the passed in types -[source] ----- -include::{dir_sact_doc_tests}/SerializationDocExamples.swift[tag=prepare_system_codable] ----- - -The `settings` argument passed to the configuration closure is an api:ActorSystemSettings[struct], which can be used to -configure various parts of the actor system. - -NOTE: While it would be nice to avoid this registration step, it does not seem possible, as `Codable` infrastructure - requires the type to be present for invoking the decoding phase. In order to be able to get the type back when - deserializing an incoming message, this mapping is used. + - + - #note: On the other hand, automatic ID assigning is inherently not entirely safe nor a good idea for building a production ready system, - so in some ways, it is good that we teach immediately about this need. Needs more docs though# - -NOTE: Serialization identifiers until 1000 are reserved for Swift Distributed Actors's internal use. #TODO sadly we have to register for every specific type... This is the correct thing to do for protocol evolution, but makes getting started harder. We can't register for "any codable" (or I can't figure out how), due to how Swift's MetaTypes work.# - -==== Step 3: Send messages as usual - -And finally, we use our messages in an actor interaction. The following function snippet is meant to reply to a query about a specific parking spot. -The sender of the query here is called a `driver` (as-in, a driver looking for a parking spot near a popular theater or cinema), -and according to some logic we reply to it with the parking spot's availability: - -.Sending messages to a (potentially) remote Actor -[source] ----- -include::{dir_sact_doc_tests}/SerializationDocExamples.swift[tag=sending_serialized_codable_messages] ----- - -What is most notable here, is that the actor code needs not concern itself _at all_ about any serialization details -of the messages and actors it interacts with. This is one of the many ways the property of _location transparency_ shows up. -Regardless of the actor being local or remote, this allows us to focus on the interactions, rather than sprinkle serialization -and/or network call concerns into the middle of our business logic. - - -WARNING: Due to this permissive style of sending messages, it technically is possible to accidentally send a message - intended only for local messaging to a remote actor, in which case such send would fail and a serialization - error would be logged. + - + - It is possible to configure `Serialization` to always serialize all messages, so you would catch such mistakes even when testing locally. <> - + - #TODO: Alternatively we could explore a mode where "all messages must be Codable" but I think this MUST be opt-in, since it makes a) getting started harder and b) not really true for internal things and c) not good if people wanted to use something else than Codable# - === Using Custom Serialization If for some reason `ProtobufRepresentable` and `Codable` are not the serialization mechanisms you want to employ to serialize your messages, @@ -208,7 +200,7 @@ NOTE: Alternatively, you may also want to explore if it is possible and/or feasi NOTE: When using `ProtobufRepresentable` or `Codable` messages, you don't need to do anything special. As long as the message containing your `ActorRef` and the `Message` type itself are registered as `ProtobufRepresentable` or `Codable` serializable, the serialization engine -will "just work", and (de)serializing refs will work transparently. This section only matters for use cases where you +will "just work", and (de-)serializing refs will work transparently. This section only matters for use cases where you implement your own serializers. The following example will serialize a simple `enum` into a simple custom format. @@ -221,6 +213,9 @@ The following example will serialize a simple `enum` into a simple custom format include::{dir_sact_doc_tests}/SerializationDocExamples.swift[tag=serialization_custom_messages] ---- +You can use the api:NonTransportableActorMessage[protocol] to provide throwing implementations of `encode(to:)`/`init(from:)` +codable functions, and instead implement the serialization using a custom serializer. + ==== Step 2: Define custom serializer .Example: Custom serializers @@ -252,10 +247,10 @@ upgrades of nodes -- as otherwise an incorrect serializer might be selected to d NOTE: Serialization identifiers until 1000 are reserved for Swift Distributed Actors's internal use. #TODO sadly we have to register for every specific type... This is the correct thing to do for protocol evolution, but makes getting started harder. We can't register for "any codable" (or I can't figure out how), due to how Swift's MetaTypes work.# -==== (De)Serializing `ActorRef` +==== (De-)Serializing `ActorRef` -Serializing and deserializing actor references is somewhat special since an actor ref has to refer to an actual "live" -entity, rather than simply be deserialized into a plain data structure. +Serializing and deserializing actor references (i.e. both `Actor` and `ActorRef` types) is somewhat special +since an actor ref has to refer to an actual "live" entity, rather than simply be deserialized into a plain data structure. In order to hook into the api:ActorSystem[class] that invokes the serializer, a special `ActorSerializationContext` is injected into serializers when they are bound to a system. If your serializer needs to handle `ActorRef`s or similar @@ -276,7 +271,7 @@ include::{dir_sact_doc_tests}/SerializationDocExamples.swift[tag=custom_actorRef Since messages may be serialized using various different techniques, the check to see if a message can be serialized has to be performed at runtime. To avoid situations where one forgets to register a serializer for a given type, it is possible to run the actor system with -the `settings.serialization.allMessages` option enabled. This causes all messages sent between actors to be serialized (even if they are not remote), +the `settings.serialization.serializeLocalMessages` option enabled. This causes all messages sent between actors to be serialized (even if they are not remote), which allows us to catch the mistake of forgetting to enable a serializer for a specific type early. TIP: This option is only intended for testing, as there is performance penalty associated with it. @@ -292,4 +287,73 @@ This will cause the _sending_ actor (or process, in case of sending messages fro Fatal error: Serialization check failed for message [NotSerializable()]:NotSerializable. Make sure this type has either a serializer registered OR is marked as `NonTransportableActorMessageNeeded`. - This check was performed since `settings.serialization.allMessages` was enabled.: file Mailbox.swift, line 260 + This check was performed since `settings.serialization.serializeLocalMessages` was enabled.: file Mailbox.swift, line 260 + +=== Internals: Serialization With Manifests + +The ActorSystem serialization infrastructure is based around so-called api:Serialization.Manifest[struct], +which carries both type "hint" as well as the serializer ID of the serializer (e.g. the _specific_ coder implementation) +that was used to serialize the payload. Thanks to these two pieces of information, it is possible for the actor system +to transparently pick the right serializer when deserializing messages on the receiving end. + +In simplified terms, messages are sent in envelopes, those envelopes contain metadata as well as the message payload itself. +The following ASCII diagram explains the general idea: + + +---- + Wire.Envelope + +-------------------------------------------+ + | Recipient | + +-------------------------------------------+ + | Metadata (e.g. trace information) | + +-------------------------------------------+ + | Manifest | + | +--------------------------------------+ | + | | serializerID, e.g. json, protobuf | | + | | typeHint? | | + | +--------------------------------------+ | + | Message (bytes...) | + +-------------------------------------------+ +---- + +The serialization into/from this wrapped wire format is performed automatically, and the `typeHint` is also able to capture +generic information of the carried message. + +It is possible to summon a type from a `Serialization.Manifest` by using the `system.serialization.summonType(manifest)` function (though statically still typed as `Any.Type`). It is also then possible to, if the type is Codable, invoke it's decoding `init(from:)` - which is what the serialization infrastructure does transparently. + +==== Advanced: Using Manifests manually + +While usually the Codable infrastructure should handle all messages automatically, you may sometimes find yourself in need of +manually carrying a "some Codable" value in your message. A typical scenario where this might happen is when implementing a +generic distributed algorithm, which will have to carry "some Codable payload that the user will provide", and the library code +should not concern itself about the details of those payloads -- maybe they will be serialized using JSON, maybe protocol buffers, or maybe using some other custom format. The library is also not in a position to determine if a type is "safe" to deserialize or not. These are all decisions that are made by end users in their systems, and configured when starting the actor system. + +We strongly recommend to sticking to the manifest and serialization infrastructure when doing so, as it allows to go +through the same "is this type trusted or not" when performing (de-)serialization of such payloads. + +Here is a simple example how one might implement a serialization of such "carry anything" while adhering to the system's +type safe-list of types: + +[source] +---- +include::{dir_sact_doc_tests}/SerializationDocExamples.swift[tag=serialize_manifest_any] +---- + + +==== A Note on Serialization.Manifest.typHint Size + +WARNING: Most of the time you should NOT be needing to drop down to this level. Make all your types Codable and let the infrastructure do the rest. We document this pattern, for those _few_ cases where it might prove beneficial _or_ necessary for other reasons. + +Automatic type hints result may result in (relatively) _large_ strings that identify the +types. Especially if payloads are highly optimized binary formats, such as protocol buffers or similar. It may happen +that type hints dominate the message size; If this is of concern to you, you can `register(MyType.self, hint: "Z")` +a type hint override, which will be used rather than the fully qualified name (or mangled name on Swift 5.3) for the type hint. + +Specialized serializers may not even need type hints _at all_ if they use some other mechanism to identify the message type, +or if they are registered with a specific serializer ID for a specific type they deserialize. + + +WARNING: **Using Swift 5.3:** The serialization is based on _mangledTypeName when used with Swift 5.3, which is very useful for prototyping, + however is also very fragile, as small changes in the types (such as a change of a type from a `struct` to a `class`, + will cause the type to be NOT the same anymore, and thus being unable to deserialize (decode) it on the receiving node + if it "still" knows about that type being a struct. This makes versioning and evolution of messages hard. diff --git a/IntegrationTests/tests_03_xpc_actorable/it_XPCActorable_echo_service/main.swift b/IntegrationTests/tests_03_xpc_actorable/it_XPCActorable_echo_service/main.swift index 983edeb46..f9e8f0098 100644 --- a/IntegrationTests/tests_03_xpc_actorable/it_XPCActorable_echo_service/main.swift +++ b/IntegrationTests/tests_03_xpc_actorable/it_XPCActorable_echo_service/main.swift @@ -26,9 +26,9 @@ let system = ActorSystem("it_XPCActorable_echo_service") { settings in settings.cluster.swim.failureDetector.pingTimeout = .seconds(3) -// settings.serialization.registerCodable(GeneratedActor.Messages.XPCEchoServiceProtocol.self, underId: 10001) -// settings.serialization.registerCodable(XPCEchoService.Message.self, underId: 10002) -// settings.serialization.registerCodable(Result.self, underId: 10003) +// settings.serialization.register(GeneratedActor.Messages.XPCEchoServiceProtocol.self, underId: 10001) +// settings.serialization.register(XPCEchoService.Message.self, underId: 10002) +// settings.serialization.register(Result.self, underId: 10003) } try! _file.append("service booted...\n") diff --git a/Package.swift b/Package.swift index 6fd5f4d38..11a491ba7 100644 --- a/Package.swift +++ b/Package.swift @@ -254,6 +254,7 @@ var dependencies: [Package.Dependency] = [ .package(url: "https://github.com/swift-server/swift-backtrace.git", from: "1.1.1"), // ~~~ SSWG APIs ~~~ + .package(url: "https://github.com/apple/swift-log.git", from: "1.0.0"), // swift-metrics 1.x and 2.x are almost API compatible, so most clients should use .package(url: "https://github.com/apple/swift-metrics.git", "1.0.0" ..< "3.0.0"), diff --git a/Samples/Sources/XPCActorCaller/main.swift b/Samples/Sources/XPCActorCaller/main.swift index d943df9a8..1b77b72a9 100644 --- a/Samples/Sources/XPCActorCaller/main.swift +++ b/Samples/Sources/XPCActorCaller/main.swift @@ -23,9 +23,9 @@ let serviceName = "com.apple.actors.xpc.GreetingsService" let system = ActorSystem("XPCActorCaller") { settings in settings.transports += .xpc -// settings.serialization.registerCodable(GeneratedActor.Messages.GreetingsService.self) -// settings.serialization.registerCodable(GreetingsServiceStub.Message.self) -// settings.serialization.registerCodable(Result.self) +// settings.serialization.register(GeneratedActor.Messages.GreetingsService.self) +// settings.serialization.register(GreetingsServiceStub.Message.self) +// settings.serialization.register(Result.self) } // TODO: we currently need a ref to the real GreetingsService... since we cannot put a Protocol.self in there... diff --git a/Samples/Sources/XPCActorServiceProvider/main.swift b/Samples/Sources/XPCActorServiceProvider/main.swift index 843001882..49850497c 100644 --- a/Samples/Sources/XPCActorServiceProvider/main.swift +++ b/Samples/Sources/XPCActorServiceProvider/main.swift @@ -22,9 +22,9 @@ let system = ActorSystem("XPCActorServiceProvider") { settings in // TODO: make this the source of "truth" what transports are available settings.transports += .xpcService -// settings.serialization.registerCodable(GeneratedActor.Messages.GreetingsService.self) -// settings.serialization.registerCodable(GreetingsServiceImpl.Message.self) -// settings.serialization.registerCodable(Result.self) +// settings.serialization.register(GeneratedActor.Messages.GreetingsService.self) +// settings.serialization.register(GreetingsServiceImpl.Message.self) +// settings.serialization.register(Result.self) } let service = try XPCActorableService(system, GreetingsServiceImpl.init) diff --git a/Sources/DistributedActors/ActorMessages.swift b/Sources/DistributedActors/ActorMessages.swift index f0a0df120..913def1f6 100644 --- a/Sources/DistributedActors/ActorMessages.swift +++ b/Sources/DistributedActors/ActorMessages.swift @@ -152,7 +152,7 @@ public struct BestEffortStringError: Error, Codable, Equatable, CustomStringConv } /// Useful error wrapper which performs an best effort Error serialization as configured by the actor system. -public struct NotTransportableAnyError: Error, NonTransportableActorMessage { +public struct NonTransportableAnyError: Error, NonTransportableActorMessage { public let failure: Error public init(_ failure: Failure) { @@ -175,23 +175,14 @@ public struct NotTransportableAnyError: Error, NonTransportableActorMessage { /// No serializer is expected to be registered for such types. /// /// - Warning: Attempting to send such message over the network will fail at runtime (and log an error or warning). -public protocol NonTransportableActorMessage: ActorMessage { - // Really what this would like to express is: - // - // func deepCopy(): Self - // - // Such that we could guarantee actors do not share state accidentally via references, - // and if we could prove a type is a value type it could safely `return self` here. - // While reference types would always need to perform a deep copy, or rely on copy on write semantics etc. - // OR if a reference type is known to be read-only / immutable, it could get away with sharing self as well perhaps? -} +public protocol NonTransportableActorMessage: ActorMessage {} extension NonTransportableActorMessage { public init(from decoder: Swift.Decoder) throws { fatalError("Attempted to decode NonTransportableActorMessage message: \(Self.self)! This should never happen.") } - public func encode(to encoder: Encoder) throws { + public func encode(to encoder: Swift.Encoder) throws { fatalError("Attempted to encode NonTransportableActorMessage message: \(Self.self)! This should never happen.") } diff --git a/Sources/DistributedActors/ActorSystem.swift b/Sources/DistributedActors/ActorSystem.swift index b5de6bff0..308bf4113 100644 --- a/Sources/DistributedActors/ActorSystem.swift +++ b/Sources/DistributedActors/ActorSystem.swift @@ -193,7 +193,7 @@ public final class ActorSystem { self._serialization = Serialization(settings: settings, system: self) } - // vvv all properties initialized, self can be shared vvv + // vvv~~~~~~~~~~~~~~~~~~~ all properties initialized, self can be shared ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~vvv // dead letters init let overrideLogger: Logger? = settings.logging.overrideLoggerFactory.map { f in f("\(ActorPath._deadLetters)") } diff --git a/Sources/DistributedActors/Cluster/Reception/OperationLogClusterReceptionist.swift b/Sources/DistributedActors/Cluster/Reception/OperationLogClusterReceptionist.swift index 7e043b68c..9e8572ab8 100644 --- a/Sources/DistributedActors/Cluster/Reception/OperationLogClusterReceptionist.swift +++ b/Sources/DistributedActors/Cluster/Reception/OperationLogClusterReceptionist.swift @@ -836,7 +836,7 @@ extension OperationLogClusterReceptionist { } public required init(from decoder: Decoder) throws { - throw SerializationError.notTransportableMessage(type: "\(Self.self)") + throw SerializationError.nonTransportableMessage(type: "\(Self.self)") } var description: String { @@ -850,7 +850,7 @@ extension OperationLogClusterReceptionist { } public required init(from decoder: Decoder) throws { - throw SerializationError.notTransportableMessage(type: "\(Self.self)") + throw SerializationError.nonTransportableMessage(type: "\(Self.self)") } var description: String { diff --git a/Sources/DistributedActors/Mailbox.swift b/Sources/DistributedActors/Mailbox.swift index db6502206..4e2214572 100644 --- a/Sources/DistributedActors/Mailbox.swift +++ b/Sources/DistributedActors/Mailbox.swift @@ -79,7 +79,7 @@ internal final class Mailbox { self.address = shell._address // TODO: not entirely happy about the added weight, but I suppose avoiding going all the way "into" the settings on each send is even worse? - self.serializeAllMessages = shell.system.settings.serialization.allMessages + self.serializeAllMessages = shell.system.settings.serialization.serializeLocalMessages } #if SACT_TESTS_LEAKS @@ -103,7 +103,7 @@ internal final class Mailbox { self.address = system.deadLetters.address // TODO: not entirely happy about the added weight, but I suppose avoiding going all the way "into" the settings on each send is even worse? - self.serializeAllMessages = system.settings.serialization.allMessages + self.serializeAllMessages = system.settings.serialization.serializeLocalMessages } @inlinable @@ -118,7 +118,7 @@ internal final class Mailbox { } catch { fatalError("Serialization check failed for message \(messageDescription) sent at \(file):\(line). " + "Make sure this type has either a serializer registered OR is marked as `NonTransportableActorMessage`. " + - "This check was performed since `settings.serialization.allMessages` was enabled.") + "This check was performed since `settings.serialization.serializeLocalMessages` was enabled.") } } diff --git a/Sources/DistributedActors/Pattern/ConvergentGossip+Serialization.swift b/Sources/DistributedActors/Pattern/ConvergentGossip+Serialization.swift index 8ea943043..eafa80910 100644 --- a/Sources/DistributedActors/Pattern/ConvergentGossip+Serialization.swift +++ b/Sources/DistributedActors/Pattern/ConvergentGossip+Serialization.swift @@ -39,7 +39,7 @@ extension ConvergentGossip.Message { try container.encode(DiscriminatorKeys.gossip, forKey: ._case) try container.encode(envelope, forKey: .gossip_envelope) default: - throw SerializationError.notTransportableMessage(type: "\(self)") + throw SerializationError.nonTransportableMessage(type: "\(self)") } } } diff --git a/Sources/DistributedActors/Protobuf/SystemMessages+Serialization.swift b/Sources/DistributedActors/Protobuf/SystemMessages+Serialization.swift index e30eafb2a..84ab0f6ea 100644 --- a/Sources/DistributedActors/Protobuf/SystemMessages+Serialization.swift +++ b/Sources/DistributedActors/Protobuf/SystemMessages+Serialization.swift @@ -95,19 +95,19 @@ extension _SystemMessage: ProtobufRepresentable { proto.payload = .terminated(terminated) case .carrySignal(let signal): - throw SerializationError.notTransportableMessage(type: "SystemMessage.carrySignal(\(signal))") + throw SerializationError.nonTransportableMessage(type: "SystemMessage.carrySignal(\(signal))") case .start: - throw SerializationError.notTransportableMessage(type: "SystemMessage.start") + throw SerializationError.nonTransportableMessage(type: "SystemMessage.start") case .nodeTerminated: - throw SerializationError.notTransportableMessage(type: "SystemMessage.addressTerminated") + throw SerializationError.nonTransportableMessage(type: "SystemMessage.addressTerminated") case .childTerminated: - throw SerializationError.notTransportableMessage(type: "SystemMessage.childTerminated") + throw SerializationError.nonTransportableMessage(type: "SystemMessage.childTerminated") case .resume: - throw SerializationError.notTransportableMessage(type: "SystemMessage.resume") + throw SerializationError.nonTransportableMessage(type: "SystemMessage.resume") case .stop: - throw SerializationError.notTransportableMessage(type: "SystemMessage.stop") + throw SerializationError.nonTransportableMessage(type: "SystemMessage.stop") case .tombstone: - throw SerializationError.notTransportableMessage(type: "SystemMessage.tombstone") + throw SerializationError.nonTransportableMessage(type: "SystemMessage.tombstone") } return proto } diff --git a/Sources/DistributedActors/Receptionist.swift b/Sources/DistributedActors/Receptionist.swift index 4297440db..a0652096b 100644 --- a/Sources/DistributedActors/Receptionist.swift +++ b/Sources/DistributedActors/Receptionist.swift @@ -87,7 +87,7 @@ public enum Receptionist { } public required init(from decoder: Decoder) throws { - throw SerializationError.notTransportableMessage(type: "") + throw SerializationError.nonTransportableMessage(type: "") } internal override var _addressableActorRef: AddressableActorRef { @@ -134,7 +134,7 @@ public enum Receptionist { } required init(from decoder: Decoder) throws { - throw SerializationError.notTransportableMessage(type: "\(Self.self)") + throw SerializationError.nonTransportableMessage(type: "\(Self.self)") } override func replyWith(_ refs: Set) { @@ -162,7 +162,7 @@ public enum Receptionist { } public required init(from decoder: Decoder) throws { - throw SerializationError.notTransportableMessage(type: "\(Self.self)") + throw SerializationError.nonTransportableMessage(type: "\(Self.self)") } internal override var _key: _RegistrationKey { @@ -430,7 +430,7 @@ public class _Lookup: ReceptionistMessage, NonTransportableActorMessage { } required init(from decoder: Decoder) throws { - throw SerializationError.notTransportableMessage(type: "\(Self.self)") + throw SerializationError.nonTransportableMessage(type: "\(Self.self)") } func replyWith(_ refs: Set) { @@ -542,7 +542,7 @@ public class _Subscribe: ReceptionistMessage, NonTransportableActorMessage { } required init(from decoder: Decoder) throws { - throw SerializationError.notTransportableMessage(type: "\(Self.self)") + throw SerializationError.nonTransportableMessage(type: "\(Self.self)") } } diff --git a/Sources/DistributedActors/Serialization/Serialization+Codable.swift b/Sources/DistributedActors/Serialization/Serialization+Codable.swift index 1e10b609b..55b3da46d 100644 --- a/Sources/DistributedActors/Serialization/Serialization+Codable.swift +++ b/Sources/DistributedActors/Serialization/Serialization+Codable.swift @@ -48,7 +48,7 @@ extension Decodable { // ==== ---------------------------------------------------------------------------------------------------------------- // MARK: Encodable + _encode(bytes:using:SomeDecoder) extensions -// TODO: once we can abstract over Coders all these could go away most likely (and accept a generic TopLevelCoder +// TODO: once we can abstract over Coders all these could go away most likely (and accept a generic TopLevelCoder) extension Encodable { func _encode(using encoder: JSONEncoder, allocator: ByteBufferAllocator) throws -> NIO.ByteBuffer { diff --git a/Sources/DistributedActors/Serialization/Serialization+Manifest.swift b/Sources/DistributedActors/Serialization/Serialization+Manifest.swift index 4629c60fe..fac9a02e7 100644 --- a/Sources/DistributedActors/Serialization/Serialization+Manifest.swift +++ b/Sources/DistributedActors/Serialization/Serialization+Manifest.swift @@ -34,8 +34,6 @@ extension Serialization { /// bytes from the message envelope size on the wire. public struct Manifest: Codable, Hashable { /// Serializer used to serialize accompanied message. - /// - /// A serializerID of zero (`0`), implies that this specific message is never intended to be serialized. public let serializerID: SerializerID /// A "hint" for the serializer what data type is serialized in the accompanying payload. @@ -131,9 +129,10 @@ extension Serialization { #endif let manifest: Manifest? - if messageType is Codable.Type { - let defaultCodableSerializerID = self.settings.defaultSerializerID - manifest = Manifest(serializerID: defaultCodableSerializerID, hint: hint) + if messageType is AnyProtobufRepresentable.Type { + manifest = Manifest(serializerID: .protobufRepresentable, hint: hint) + } else if messageType is Codable.Type { + manifest = Manifest(serializerID: self.settings.defaultSerializerID, hint: hint) } else if messageType is NonTransportableActorMessage.Type { manifest = Manifest(serializerID: .doNotSerialize, hint: nil) } else { diff --git a/Sources/DistributedActors/Serialization/Serialization+SerializerID.swift b/Sources/DistributedActors/Serialization/Serialization+SerializerID.swift index b4c35aaeb..74d385f73 100644 --- a/Sources/DistributedActors/Serialization/Serialization+SerializerID.swift +++ b/Sources/DistributedActors/Serialization/Serialization+SerializerID.swift @@ -31,7 +31,7 @@ extension Serialization { switch self.value { case SerializerID.doNotSerialize.value: return "serializerID:doNotSerialize(\(self.value))" - case SerializerID.specialized.value: + case SerializerID.specializedWithTypeHint.value: return "serializerID:specialized(\(self.value))" case SerializerID.foundationJSON.value: return "serializerID:jsonCodable(\(self.value))" @@ -63,7 +63,7 @@ extension Serialization.SerializerID { // ~~~~~~~~~~~~~~~~ general purpose serializer ids ~~~~~~~~~~~~~~~~ public static let doNotSerialize: SerializerID = 0 - public static let specialized: SerializerID = 1 + public static let specializedWithTypeHint: SerializerID = 1 public static let foundationJSON: SerializerID = 2 // public static let foundationPropertyList: SerializerID = 3 // TODO: https://github.com/apple/swift-distributed-actors/issues/513 public static let protobufRepresentable: SerializerID = 4 diff --git a/Sources/DistributedActors/Serialization/Serialization+Serializers.swift b/Sources/DistributedActors/Serialization/Serialization+Serializers.swift index 704534e69..9821f29fb 100644 --- a/Sources/DistributedActors/Serialization/Serialization+Serializers.swift +++ b/Sources/DistributedActors/Serialization/Serialization+Serializers.swift @@ -78,10 +78,10 @@ extension Serializer: AnySerializer { } // ==== ---------------------------------------------------------------------------------------------------------------- -// MARK: NotTransportableSerializer +// MARK: NonTransportableSerializer /// Nope, as opposed to Noop -internal class NotTransportableSerializer: Serializer { +internal class NonTransportableSerializer: Serializer { override func serialize(_ message: Message) throws -> ByteBuffer { throw SerializationError.unableToSerialize(hint: "\(Self.self): \(Message.self)") } diff --git a/Sources/DistributedActors/Serialization/Serialization+Settings.swift b/Sources/DistributedActors/Serialization/Serialization+Settings.swift index 07e355af1..78dc2fc28 100644 --- a/Sources/DistributedActors/Serialization/Serialization+Settings.swift +++ b/Sources/DistributedActors/Serialization/Serialization+Settings.swift @@ -34,10 +34,34 @@ extension Serialization { .init() } + /// When `true`, all messages are allowed to be sent (serialized, and deserialized) regardless if they were + /// registered with serialization or not. While this setting is true, the system will log a warning about each + /// message type when it is first encountered during the systems operation, and it will suggest registering that type. + /// This way one can use this setting in local debugging and quick iteration, and then easily register all necessary types + /// when deploying to production. + /// + /// - Warning: Do not set this value to true in production deployments, as it could be used send and deserialize any codable type + /// and the serialization infrastructure would attempt deserializing it, potentially opening up for security risks. + // TODO: We are using an internal function here to allow us to automatically enable the more strict mode in release builds. + public var insecureSerializeNotRegisteredMessages: Bool = _isDebugAssertConfiguration() + /// Serializes all messages, also when passed only locally between actors. /// - /// Use this option to test that all messages you expected to - public var allMessages: Bool = false + /// This option to ensure no reference types are "leaked" through message passing, + /// as messages now will always be passed through serialization accidental sharing of + /// mutable state (which would have been unsafe) can be avoided by the serialization round trip. + /// + /// - Warning: Do not use this setting in production settings if you care for performance however, + /// as it implies needless serialization roundtrips on every single message send on the entire system (!). + public var serializeLocalMessages: Bool = false + + /// Configures which `Codable` serializer (`Encoder` / `Decoder` pair) should be used whenever a + /// a message is sent however the type does not have a specific serializer requirement configured (via `register` calls). + /// + /// // TODO: This should default to some nice binary format rather than JSON. + /// + /// - Note: Affects only _outbound_ messages which are `Codable`. + public var defaultSerializerID: Serialization.SerializerID = .foundationJSON /// `UniqueNode` to be included in actor addresses when serializing them. /// By default this should be equal to the exposed node of the actor system. @@ -46,12 +70,8 @@ extension Serialization { /// as it is not useful to render any address for actors which shall never be reached remotely. /// /// This is set automatically when modifying the systems cluster settings. - internal var localNode: UniqueNode = .init(systemName: "", host: "127.0.0.1", port: 7337, nid: NodeID(0)) - - /// Configures which `Codable` serializer should be used whenever a - /// - /// - Note: Affects only _outbound_ messages which are `Codable`. - public var defaultSerializerID: Serialization.SerializerID = .foundationJSON + internal var localNode: UniqueNode = + .init(systemName: "", host: "127.0.0.1", port: 7337, nid: NodeID(0)) /// Applied before automatically selecting a serializer based on manifest. /// Allows to deserialize incoming messages when "the same" message is now represented on this system differently. @@ -91,26 +111,31 @@ extension Serialization.Settings { /// This can be used to "force" a specific serializer be used for a message type, /// regardless if it is codable or not. @discardableResult - public mutating func registerManifest( + public mutating func register( _ type: Message.Type, hint hintOverride: String? = nil, - serializer overrideSerializerID: SerializerID? + serializerID overrideSerializerID: SerializerID? = nil ) -> Manifest { // FIXME: THIS IS A WORKAROUND UNTIL WE CAN GET MANGLED NAMES let hint = hintOverride ?? _typeName(type) // FIXME: _mangledTypeName https://github.com/apple/swift/pull/30318 - let serializerID = overrideSerializerID ?? self.defaultSerializerID // TODO: We could do educated guess work here -- if a type is protobuf representable, that's the coding we want - switch serializerID { - case .protobufRepresentable: + // TODO: add test for sending raw SwiftProtobuf.Message + if overrideSerializerID == SerializerID.protobufRepresentable { precondition( type is AnyProtobufRepresentable.Type || type is SwiftProtobuf.Message.Type, """ Attempted to register \(String(reflecting: type)) as \ - serializable using \(SerializerID.protobufRepresentable) yet the type does NOT conform to ProtobufRepresentable or SwiftProtobuf.Message + serializable using \(reflecting: overrideSerializerID), \ + yet the type does NOT conform to ProtobufRepresentable or SwiftProtobuf.Message """ ) - default: - () // OK + } + + let serializerID: SerializerID + if Message.self is AnyProtobufRepresentable.Type { + serializerID = .protobufRepresentable + } else { + serializerID = overrideSerializerID ?? self.defaultSerializerID } let manifest = Manifest(serializerID: serializerID, hint: hint) @@ -125,9 +150,9 @@ extension Serialization.Settings { /// /// This manifest will NOT be used when _sending_ messages of the `Message` type. @discardableResult - public mutating func registerInboundManifest( + public mutating func registerInbound( _ type: Message.Type, hint hintOverride: String? = nil, - serializer overrideSerializerID: SerializerID? + serializerID overrideSerializerID: SerializerID? = nil ) -> Manifest { // FIXME: THIS IS A WORKAROUND UNTIL WE CAN GET MANGLED NAMES https://github.com/apple/swift/pull/30318 let hint = hintOverride ?? _typeName(type) // FIXME: _mangledTypeName https://github.com/apple/swift/pull/30318 @@ -142,73 +167,22 @@ extension Serialization.Settings { } // ==== ---------------------------------------------------------------------------------------------------------------- -// MARK: Serialization: Codable manifest and serializer registration - -extension Serialization.Settings { - /// Eagerly register a `Codable` message type to be used with a specific serializer. - /// - /// By doing this before system startup you can ensure a specific serializer is used for those messages. - /// Make sure tha other nodes in the system are configured the same way though. - public mutating func registerCodable( - _ type: Message.Type, hint hintOverride: String? = nil, - serializer serializerOverride: SerializerID? = nil - ) { - let hint = hintOverride ?? _typeName(type) // FIXME: _mangledTypeName https://github.com/apple/swift/pull/30318 - let serializerID = serializerOverride ?? self.defaultSerializerID - let manifest = Manifest(serializerID: serializerID, hint: hint) - - self.typeToManifestRegistry[.init(type)] = manifest - self.manifest2TypeRegistry[manifest] = type - } -} - -// ==== ---------------------------------------------------------------------------------------------------------------- -// MARK: Serialization: ProtobufRepresentable +// MARK: Serialization: Specialized extension Serialization.Settings { - /// Register a type to be serialized using Google Protocol Buffers. - /// - /// The type should conform to `ProtobufRepresentable`, in order to instruct the serializer infrastructure - /// how to de/encode it from its protobuf representation. - public mutating func registerProtobufRepresentable( - _ type: Message.Type - ) { - // 1. register the specific to this type serializer (maker) - self.registerSpecializedSerializer(type, serializer: .protobufRepresentable) { allocator in - ProtobufSerializer(allocator: allocator) // FIXME: should be able to avoid registering all together - } - - // 2. register manifest pointing to that specialized serializer - let manifest = self.getSpecializedOrRegisterManifest(type, serializerID: .protobufRepresentable) - self.typeToManifestRegistry[.init(type)] = manifest - self.manifest2TypeRegistry[manifest] = type - } - - // Internal since we want to touch only internal types and not be forced to make the public. - internal mutating func _registerInternalProtobufRepresentable( - _ type: Message.Type - ) { - // 1. register the specific to this type serializer (maker) - self.registerSpecializedSerializer(type, serializer: .protobufRepresentable) { allocator in - InternalProtobufSerializer(allocator: allocator) // FIXME: should be able to avoid registering all together - } - - // 2. register manifest pointing to that specialized serializer - let manifest = self.getSpecializedOrRegisterManifest(type, serializerID: .protobufRepresentable) - self.typeToManifestRegistry[.init(type)] = manifest - self.manifest2TypeRegistry[manifest] = type - } - - // TODO: impl not entirely perfect yet... more tests /// Register a specialized serializer for a specific `Serialization.Manifest`. - public mutating func registerSpecializedSerializer( + public mutating func registerSpecializedSerializer( _ type: Message.Type, hint hintOverride: String? = nil, - serializer: SerializerID, + serializerID: SerializerID, makeSerializer: @escaping (NIO.ByteBufferAllocator) -> Serializer ) { + precondition( + serializerID == .specializedWithTypeHint || serializerID > 16, + "Specialized serializerID MUST exactly `1` or be `> 16`, since IDs until 16 are reserved for general purpose serializers" + ) // FIXME: THIS IS A WORKAROUND UNTIL WE CAN GET MANGLED NAMES https://github.com/apple/swift/pull/30318 let hint = hintOverride ?? _typeName(type) // FIXME: _mangledTypeName https://github.com/apple/swift/pull/30318 - let manifest = Serialization.Manifest(serializerID: serializer, hint: hint) + let manifest = Serialization.Manifest(serializerID: serializerID, hint: hint) self.specializedSerializerMakers[manifest] = { allocator in makeSerializer(allocator).asAnySerializer @@ -220,6 +194,6 @@ extension Serialization.Settings { serializerID: Serialization.SerializerID ) -> Serialization.Manifest { self.typeToManifestRegistry[.init(type)] ?? - self.registerManifest(type, serializer: serializerID) + self.register(type, serializerID: serializerID) } } diff --git a/Sources/DistributedActors/Serialization/Serialization.swift b/Sources/DistributedActors/Serialization/Serialization.swift index 97ae0ea1e..30263674a 100644 --- a/Sources/DistributedActors/Serialization/Serialization.swift +++ b/Sources/DistributedActors/Serialization/Serialization.swift @@ -70,104 +70,104 @@ public class Serialization { var settings = systemSettings.serialization // ==== Declare mangled names of some known popular types // TODO: hardcoded mangled name until we have _mangledTypeName - settings.registerCodable(Bool.self, hint: "b", serializer: .specialized) - settings.registerSpecializedSerializer(Bool.self, hint: "b", serializer: .specialized) { allocator in + settings.register(Bool.self, hint: "b", serializerID: .specializedWithTypeHint) + settings.registerSpecializedSerializer(Bool.self, hint: "b", serializerID: .specializedWithTypeHint) { allocator in BoolSerializer(allocator) } // harder since no direct mapping to write... onto a byte buffer - // settings.registerCodable(Float.self, hint: "f", serializer: .specialized) - // settings.registerCodable(Float32.self, hint: "f", serializer: .specialized) - // settings.registerCodable(Float64.self, hint: "d", serializer: .specialized) + // settings.register(Float.self, hint: "f", serializerID: .specializedWithTypeHint) + // settings.register(Float32.self, hint: "f", serializerID: .specializedWithTypeHint) + // settings.register(Float64.self, hint: "d", serializerID: .specializedWithTypeHint) - settings.registerCodable(Int.self, hint: "i", serializer: .specialized) - settings.registerSpecializedSerializer(Int.self, hint: "i", serializer: .specialized) { allocator in + settings.register(Int.self, hint: "i", serializerID: .specializedWithTypeHint) + settings.registerSpecializedSerializer(Int.self, hint: "i", serializerID: .specializedWithTypeHint) { allocator in IntegerSerializer(Int.self, allocator) } - settings.registerCodable(UInt.self, hint: "u", serializer: .specialized) - settings.registerSpecializedSerializer(UInt.self, hint: "u", serializer: .specialized) { allocator in + settings.register(UInt.self, hint: "u", serializerID: .specializedWithTypeHint) + settings.registerSpecializedSerializer(UInt.self, hint: "u", serializerID: .specializedWithTypeHint) { allocator in IntegerSerializer(UInt.self, allocator) } - settings.registerCodable(Int64.self, hint: "i64", serializer: .specialized) - settings.registerSpecializedSerializer(Int64.self, hint: "i64", serializer: .specialized) { allocator in + settings.register(Int64.self, hint: "i64", serializerID: .specializedWithTypeHint) + settings.registerSpecializedSerializer(Int64.self, hint: "i64", serializerID: .specializedWithTypeHint) { allocator in IntegerSerializer(Int64.self, allocator) } - settings.registerCodable(UInt64.self, hint: "u64", serializer: .specialized) - settings.registerSpecializedSerializer(UInt64.self, hint: "u64", serializer: .specialized) { allocator in + settings.register(UInt64.self, hint: "u64", serializerID: .specializedWithTypeHint) + settings.registerSpecializedSerializer(UInt64.self, hint: "u64", serializerID: .specializedWithTypeHint) { allocator in IntegerSerializer(UInt64.self, allocator) } - settings.registerCodable(String.self, hint: "S", serializer: .specialized) - settings.registerSpecializedSerializer(String.self, hint: "S", serializer: .specialized) { allocator in + settings.register(String.self, hint: "S", serializerID: .specializedWithTypeHint) + settings.registerSpecializedSerializer(String.self, hint: "S", serializerID: .specializedWithTypeHint) { allocator in StringSerializer(allocator) } - settings.registerCodable(String?.self, hint: "qS") - settings.registerCodable(Int?.self, hint: "qI") + settings.register(String?.self, hint: "qS") + settings.register(Int?.self, hint: "qI") // ==== Declare some system messages to be handled with specialized serializers: // system messages - settings.registerProtobufRepresentable(_SystemMessage.self) - settings._registerInternalProtobufRepresentable(_SystemMessage.ACK.self) - settings._registerInternalProtobufRepresentable(_SystemMessage.NACK.self) - settings._registerInternalProtobufRepresentable(SystemMessageEnvelope.self) + settings.register(_SystemMessage.self) + settings.register(_SystemMessage.ACK.self) + settings.register(_SystemMessage.NACK.self) + settings.register(SystemMessageEnvelope.self) // cluster - settings._registerInternalProtobufRepresentable(ClusterShell.Message.self) - settings.registerProtobufRepresentable(Cluster.Event.self) - settings.registerCodable(ConvergentGossip.Message.self) // TODO: can be removed once https://github.com/apple/swift/pull/30318 lands + settings.register(ClusterShell.Message.self) + settings.register(Cluster.Event.self) + settings.register(ConvergentGossip.Message.self) // TODO: can be removed once https://github.com/apple/swift/pull/30318 lands // receptionist needs some special casing // TODO: document how to deal with `protocol` message accepting actors, those should be very rare. // TODO: do we HAVE to do this in the Receptionist? - settings.registerManifest(Receptionist.Message.self, serializer: .doNotSerialize) - settings.registerCodable(OperationLogClusterReceptionist.AckOps.self) // TODO: can be removed once https://github.com/apple/swift/pull/30318 lands + settings.register(Receptionist.Message.self, serializerID: .doNotSerialize) + settings.register(OperationLogClusterReceptionist.AckOps.self) // TODO: can be removed once https://github.com/apple/swift/pull/30318 lands // FIXME: This will go away once https://github.com/apple/swift/pull/30318 is merged and we can rely on summoning types - settings.registerCodable(OperationLogClusterReceptionist.PushOps.self) // TODO: can be removed once https://github.com/apple/swift/pull/30318 lands - settings.registerInboundManifest( + settings.register(OperationLogClusterReceptionist.PushOps.self) // TODO: can be removed once https://github.com/apple/swift/pull/30318 lands + settings.registerInbound( OperationLogClusterReceptionist.PushOps.self, - hint: "DistributedActors.\(OperationLogClusterReceptionist.PushOps.self)", serializer: .default + hint: "DistributedActors.\(OperationLogClusterReceptionist.PushOps.self)", serializerID: .default ) // FIXME: This will go away once https://github.com/apple/swift/pull/30318 is merged and we can rely on summoning types - settings.registerInboundManifest(OperationLogClusterReceptionist.AckOps.self, hint: "ReceptionistMessage", serializer: .default) - settings.registerInboundManifest( + settings.registerInbound(OperationLogClusterReceptionist.AckOps.self, hint: "ReceptionistMessage", serializerID: .default) + settings.registerInbound( OperationLogClusterReceptionist.AckOps.self, - hint: "DistributedActors.\(OperationLogClusterReceptionist.AckOps.self)", serializer: .default + hint: "DistributedActors.\(OperationLogClusterReceptionist.AckOps.self)", serializerID: .default ) // swim failure detector - settings._registerInternalProtobufRepresentable(SWIM.Message.self) - settings._registerInternalProtobufRepresentable(SWIM.RemoteMessage.self) - settings._registerInternalProtobufRepresentable(SWIM.PingResponse.self) + settings.register(SWIM.Message.self) + settings.register(SWIM.RemoteMessage.self) + settings.register(SWIM.PingResponse.self) // TODO: Allow plugins to register types...? - settings.registerManifest(ActorAddress.self, serializer: .protobufRepresentable) - settings.registerManifest(ReplicaID.self, serializer: .foundationJSON) - settings.registerManifest(VersionDot.self, serializer: .protobufRepresentable) - settings.registerManifest(VersionVector.self, serializer: .protobufRepresentable) + settings.register(ActorAddress.self, serializerID: .protobufRepresentable) + settings.register(ReplicaID.self, serializerID: .foundationJSON) + settings.register(VersionDot.self, serializerID: .protobufRepresentable) + settings.register(VersionVector.self, serializerID: .protobufRepresentable) // crdts // TODO: all this registering will go away with _mangledTypeName - settings.registerManifest(CRDT.Identity.self, serializer: .protobufRepresentable) - settings.registerManifest(CRDT.VersionedContainer.self, serializer: .protobufRepresentable) - settings.registerManifest(CRDT.VersionContext.self, serializer: .protobufRepresentable) - settings.registerManifest(CRDT.VersionedContainerDelta.self, serializer: .protobufRepresentable) - settings.registerManifest(CRDT.VersionedContainerDelta.self, serializer: .protobufRepresentable) - settings.registerManifest(CRDT.Replicator.Message.self, serializer: .protobufRepresentable) - settings.registerManifest(CRDT.Envelope.self, serializer: .protobufRepresentable) - settings.registerManifest(CRDT.Replicator.RemoteCommand.WriteResult.self, serializer: .protobufRepresentable) - settings.registerManifest(CRDT.Replicator.RemoteCommand.ReadResult.self, serializer: .protobufRepresentable) - settings.registerManifest(CRDT.Replicator.RemoteCommand.DeleteResult.self, serializer: .protobufRepresentable) - settings.registerManifest(CRDT.GCounter.self, serializer: .protobufRepresentable) - settings.registerManifest(CRDT.GCounterDelta.self, serializer: .protobufRepresentable) - settings.registerManifest(CRDT.ORSet.self, serializer: .protobufRepresentable) - settings.registerManifest(CRDT.ORSet.self, serializer: .protobufRepresentable) - // settings.registerManifest(AnyDeltaCRDT.self, serializer: ReservedID.CRDTDeltaBox) // FIXME: so we cannot test the CRDT.Envelope+SerializationTests + settings.register(CRDT.Identity.self, serializerID: .protobufRepresentable) + settings.register(CRDT.VersionedContainer.self, serializerID: .protobufRepresentable) + settings.register(CRDT.VersionContext.self, serializerID: .protobufRepresentable) + settings.register(CRDT.VersionedContainerDelta.self, serializerID: .protobufRepresentable) + settings.register(CRDT.VersionedContainerDelta.self, serializerID: .protobufRepresentable) + settings.register(CRDT.Replicator.Message.self, serializerID: .protobufRepresentable) + settings.register(CRDT.Envelope.self, serializerID: .protobufRepresentable) + settings.register(CRDT.Replicator.RemoteCommand.WriteResult.self, serializerID: .protobufRepresentable) + settings.register(CRDT.Replicator.RemoteCommand.ReadResult.self, serializerID: .protobufRepresentable) + settings.register(CRDT.Replicator.RemoteCommand.DeleteResult.self, serializerID: .protobufRepresentable) + settings.register(CRDT.GCounter.self, serializerID: .protobufRepresentable) + settings.register(CRDT.GCounterDelta.self, serializerID: .protobufRepresentable) + settings.register(CRDT.ORSet.self, serializerID: .protobufRepresentable) + settings.register(CRDT.ORSet.self, serializerID: .protobufRepresentable) + // settings.register(AnyDeltaCRDT.self, serializerID:ReservedID.CRDTDeltaBox) // FIXME: so we cannot test the CRDT.Envelope+SerializationTests // errors - settings.registerCodable(ErrorEnvelope.self) // TODO: can be removed once https://github.com/apple/swift/pull/30318 lands - settings.registerCodable(BestEffortStringError.self) // TODO: can be removed once https://github.com/apple/swift/pull/30318 lands + settings.register(ErrorEnvelope.self) // TODO: can be removed once https://github.com/apple/swift/pull/30318 lands + settings.register(BestEffortStringError.self) // TODO: can be removed once https://github.com/apple/swift/pull/30318 lands self.settings = settings self.metrics = system.metrics @@ -237,7 +237,11 @@ extension Serialization { #endif } - public func _ensureSerializer(_ type: Message.Type, file: String = #file, line: UInt = #line) throws { + /// Ensures the `Message` will be able to be serialized, using either a specific or default serializer. + /// + /// By default, if in `insecureSerializeNotRegisteredMessages` mode, this logs warnings and allows all messages + /// to be serialized. If the setting `insecureSerializeNotRegisteredMessages` is `false`, then + public func _ensureSerializer(_ type: Message.Type, file: String = #file, line: UInt = #line) throws { let oid = ObjectIdentifier(type) // 1. check if this type already has a serializer registered, bail out quickly if so @@ -259,7 +263,6 @@ extension Serialization { // 2.3. create and store the appropriate serializer do { - traceLog_Serialization("Registered [\(manifest)] for [\(reflecting: type)]") self._serializers[oid] = try self.makeCodableSerializer(type, manifest: manifest) } catch SerializationError.noNeedToEnsureSerializer { // some types are specifically marked as "do not serialize" and we should ignore failures @@ -270,6 +273,17 @@ extension Serialization { // all other errors are real and should be escalated throw error } + + traceLog_Serialization("Registered [\(manifest)] for [\(reflecting: type)]") + // TODO: decide if we log or crash when new things reg ensured during runtime + // FIXME: https://github.com/apple/swift-distributed-actors/issues/552 +// if self.settings.insecureSerializeNotRegisteredMessages { +// self.log.warning(""" +// Type [\(String(reflecting: type))] was not registered with Serialization, \ +// sending this message will cause it to be dropped in release mode! Use: \ +// settings.serialization.register(\(type).self) to avoid this in production. +// """) +// } } } @@ -278,7 +292,7 @@ extension Serialization { case .doNotSerialize: throw SerializationError.noNeedToEnsureSerializer - case Serialization.SerializerID.specialized: + case Serialization.SerializerID.specializedWithTypeHint: guard let make = self.settings.specializedSerializerMakers[manifest] else { throw SerializationError.unableToMakeSerializer(hint: "Type: \(String(reflecting: type)), Manifest: \(manifest), Specialized serializer makers: \(self.settings.specializedSerializerMakers)") } @@ -347,7 +361,7 @@ extension Serialization { } else if let encodableMessage = messageType as? Encodable { // TODO: we need to be able to abstract over Coders to collapse this into "giveMeACoder().encode()" switch manifest.serializerID { - case .specialized: + case .specializedWithTypeHint: throw SerializationError.unableToMakeSerializer( hint: """ @@ -439,7 +453,7 @@ extension Serialization { do { // Manifest type may be used to summon specific instances of types from the manifest // even if the expected type is some `Outer` type (e.g. when we sent a sub class). - let manifestMessageType = try self.summonType(from: manifest) + let manifestMessageType: Any.Type = try self.summonType(from: manifest) let manifestMessageTypeID = ObjectIdentifier(manifestMessageType) let messageTypeID = ObjectIdentifier(manifestMessageType) @@ -451,7 +465,7 @@ extension Serialization { } else if let decodableMessageType = manifestMessageType as? Decodable.Type { // TODO: we need to be able to abstract over Coders to collapse this into "giveMeACoder().decode()" switch manifest.serializerID { - case .specialized: + case .specializedWithTypeHint: throw SerializationError.unableToMakeSerializer( hint: """ @@ -649,7 +663,7 @@ public enum SerializationError: Error { case serializationError(_: Error, file: String, line: UInt) // --- registration errors --- - case alreadyDefined(hint: String, serializerID: Serialization.SerializerID, serializer: AnySerializer?) + case alreadyDefined(hint: String, serializerID: Serialization.SerializerID, serializerID: AnySerializer?) case reservedSerializerID(hint: String) // --- lookup errors --- @@ -675,7 +689,7 @@ public enum SerializationError: Error { case unknownEnumValue(Int) // --- illegal errors --- - case notTransportableMessage(type: String) + case nonTransportableMessage(type: String) case unableToMakeSerializer(hint: String) case unableToSerialize(hint: String) diff --git a/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift b/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift index 79a2fd3be..5b7e4013c 100644 --- a/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift +++ b/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift @@ -28,21 +28,21 @@ final class ActorSingletonPluginClusteredTests: ClusteredNodesTestBase { settings.cluster.node.port = 7111 settings.cluster.autoLeaderElection = .lowestReachable(minNumberOfMembers: 3) - settings.serialization.registerCodable(GreeterSingleton.Message.self) + settings.serialization.register(GreeterSingleton.Message.self) } let second = self.setUpNode("second") { settings in settings += ActorSingletonPlugin() settings.cluster.node.port = 8222 settings.cluster.autoLeaderElection = .lowestReachable(minNumberOfMembers: 3) - settings.serialization.registerCodable(GreeterSingleton.Message.self) + settings.serialization.register(GreeterSingleton.Message.self) } let third = self.setUpNode("third") { settings in settings += ActorSingletonPlugin() settings.cluster.node.port = 9333 settings.cluster.autoLeaderElection = .lowestReachable(minNumberOfMembers: 3) - settings.serialization.registerCodable(GreeterSingleton.Message.self) + settings.serialization.register(GreeterSingleton.Message.self) } // Bring up `ActorSingletonProxy` before setting up cluster (https://github.com/apple/swift-distributed-actors/issues/463) @@ -81,21 +81,21 @@ final class ActorSingletonPluginClusteredTests: ClusteredNodesTestBase { settings.cluster.node.port = 7111 settings.cluster.autoLeaderElection = .lowestReachable(minNumberOfMembers: 3) - settings.serialization.registerCodable(GreeterSingleton.Message.self) + settings.serialization.register(GreeterSingleton.Message.self) } let second = self.setUpNode("second") { settings in settings += ActorSingletonPlugin() settings.cluster.node.port = 8222 settings.cluster.autoLeaderElection = .lowestReachable(minNumberOfMembers: 3) - settings.serialization.registerCodable(GreeterSingleton.Message.self) + settings.serialization.register(GreeterSingleton.Message.self) } let third = self.setUpNode("third") { settings in settings += ActorSingletonPlugin() settings.cluster.node.port = 9333 settings.cluster.autoLeaderElection = .lowestReachable(minNumberOfMembers: 3) - settings.serialization.registerCodable(GreeterSingleton.Message.self) + settings.serialization.register(GreeterSingleton.Message.self) } // No leader so singleton is not available, messages sent should be stashed @@ -143,28 +143,28 @@ final class ActorSingletonPluginClusteredTests: ClusteredNodesTestBase { settings.cluster.node.port = 7111 settings.cluster.autoLeaderElection = .lowestReachable(minNumberOfMembers: 3) - settings.serialization.registerCodable(GreeterSingleton.Message.self) + settings.serialization.register(GreeterSingleton.Message.self) } let second = self.setUpNode("second") { settings in settings += ActorSingletonPlugin() settings.cluster.node.port = 8222 settings.cluster.autoLeaderElection = .lowestReachable(minNumberOfMembers: 3) - settings.serialization.registerCodable(GreeterSingleton.Message.self) + settings.serialization.register(GreeterSingleton.Message.self) } let third = self.setUpNode("third") { settings in settings += ActorSingletonPlugin() settings.cluster.node.port = 9333 settings.cluster.autoLeaderElection = .lowestReachable(minNumberOfMembers: 3) - settings.serialization.registerCodable(GreeterSingleton.Message.self) + settings.serialization.register(GreeterSingleton.Message.self) } let fourth = self.setUpNode("fourth") { settings in settings += ActorSingletonPlugin() settings.cluster.node.port = 7444 settings.cluster.autoLeaderElection = .lowestReachable(minNumberOfMembers: 3) - settings.serialization.registerCodable(GreeterSingleton.Message.self) + settings.serialization.register(GreeterSingleton.Message.self) } // Bring up `ActorSingletonProxy` before setting up cluster (https://github.com/apple/swift-distributed-actors/issues/463) diff --git a/Tests/DistributedActorsDocumentationTests/ActorDocExamples.swift b/Tests/DistributedActorsDocumentationTests/ActorDocExamples.swift index 14bc51cc2..382210d27 100644 --- a/Tests/DistributedActorsDocumentationTests/ActorDocExamples.swift +++ b/Tests/DistributedActorsDocumentationTests/ActorDocExamples.swift @@ -268,7 +268,7 @@ class ActorDocExamples: XCTestCase { let system = ActorSystem("ExampleSystem") // tag::ask_outside[] - struct Hello: ActorMessage { + struct Hello: Codable { let name: String let replyTo: ActorRef } @@ -292,7 +292,7 @@ class ActorDocExamples: XCTestCase { let system = ActorSystem("ExampleSystem") // tag::ask_inside[] - struct Hello: ActorMessage { + struct Hello: Codable { let name: String let replyTo: ActorRef } @@ -331,7 +331,7 @@ class ActorDocExamples: XCTestCase { let ref: ActorRef! = nil // tag::eventStream[] - enum Event: String, ActorMessage { + enum Event: String, Codable { case eventOne case eventTwo } @@ -362,7 +362,7 @@ struct ExampleWorker { internal static var props: Props = Props().dispatcher(.pinnedThread) } -enum WorkerMessages: String, ActorMessage { +enum WorkerMessages: String, Codable { case something } diff --git a/Tests/DistributedActorsDocumentationTests/Actorable/ActorableDocExamples.swift b/Tests/DistributedActorsDocumentationTests/Actorable/ActorableDocExamples.swift index 2797cf120..1d6099a71 100644 --- a/Tests/DistributedActorsDocumentationTests/Actorable/ActorableDocExamples.swift +++ b/Tests/DistributedActorsDocumentationTests/Actorable/ActorableDocExamples.swift @@ -138,9 +138,9 @@ struct AllInOneMachine: Actorable, CoffeeMachine, Diagnostics { // <5> } // end::compose_protocols_1[] -public struct Tea: ActorMessage {} +public struct Tea: Codable {} -public struct Coffee: ActorMessage {} +public struct Coffee: Codable {} class UsingAllInOneMachine { func run() throws { diff --git a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/AccessControl+GenActor.swift b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/AccessControl+GenActor.swift index 9e0e9ef11..33a26bf91 100644 --- a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/AccessControl+GenActor.swift +++ b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/AccessControl+GenActor.swift @@ -31,7 +31,7 @@ import XCTest /// DO NOT EDIT: Generated AccessControl messages extension AccessControl { - public enum Message: ActorMessage { + public enum Message: Codable { case greetPublicly case greetInternal } diff --git a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/AllInOneMachine+GenActor.swift b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/AllInOneMachine+GenActor.swift index 9fa348422..6a92a6700 100644 --- a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/AllInOneMachine+GenActor.swift +++ b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/AllInOneMachine+GenActor.swift @@ -31,7 +31,7 @@ import XCTest /// DO NOT EDIT: Generated AllInOneMachine messages extension AllInOneMachine { - public enum Message: ActorMessage { + public enum Message: Codable { case clean case coffeeMachine(/*TODO: MODULE.*/GeneratedActor.Messages.CoffeeMachine) case diagnostics(/*TODO: MODULE.*/GeneratedActor.Messages.Diagnostics) diff --git a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/CoffeeMachine+GenActor.swift b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/CoffeeMachine+GenActor.swift index fe65cc1a6..0f5a07962 100644 --- a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/CoffeeMachine+GenActor.swift +++ b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/CoffeeMachine+GenActor.swift @@ -29,7 +29,7 @@ import XCTest // MARK: DO NOT EDIT: Generated CoffeeMachine messages extension GeneratedActor.Messages { - public enum CoffeeMachine: ActorMessage { + public enum CoffeeMachine: Codable { case makeCoffee(_replyTo: ActorRef) } } diff --git a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/ContextGreeter+GenActor.swift b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/ContextGreeter+GenActor.swift index ad7c55519..30f33dadc 100644 --- a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/ContextGreeter+GenActor.swift +++ b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/ContextGreeter+GenActor.swift @@ -31,7 +31,7 @@ import XCTest /// DO NOT EDIT: Generated ContextGreeter messages extension ContextGreeter { - public enum Message: ActorMessage { + public enum Message: Codable { case greet(name: String, _replyTo: ActorRef) } diff --git a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/Diagnostics+GenActor.swift b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/Diagnostics+GenActor.swift index 7eeaf64f6..683fe9522 100644 --- a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/Diagnostics+GenActor.swift +++ b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/Diagnostics+GenActor.swift @@ -29,7 +29,7 @@ import XCTest // MARK: DO NOT EDIT: Generated Diagnostics messages extension GeneratedActor.Messages { - public enum Diagnostics: ActorMessage { + public enum Diagnostics: Codable { case printDiagnostics } } diff --git a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/DontConformMessageToCodable+GenActor.swift b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/DontConformMessageToCodable+GenActor.swift index 9d9664a67..6957bac03 100644 --- a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/DontConformMessageToCodable+GenActor.swift +++ b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/DontConformMessageToCodable+GenActor.swift @@ -31,7 +31,7 @@ import XCTest /// DO NOT EDIT: Generated DontConformMessageToCodable messages extension DontConformMessageToCodable { - public enum Message: ActorMessage { + public enum Message: Codable { case echo(text: String, _replyTo: ActorRef) } diff --git a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/GreetMe+GenActor.swift b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/GreetMe+GenActor.swift index 71e00e673..675fdbf85 100644 --- a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/GreetMe+GenActor.swift +++ b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/GreetMe+GenActor.swift @@ -26,7 +26,7 @@ import XCTest /// DO NOT EDIT: Generated GreetMe messages extension GreetMe { - public enum Message: ActorMessage { + public enum Message: Codable { case hello(greeting: String) } diff --git a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/GreetMeGreeter+GenActor.swift b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/GreetMeGreeter+GenActor.swift index 3f10e2b84..9c7c46cab 100644 --- a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/GreetMeGreeter+GenActor.swift +++ b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/GreetMeGreeter+GenActor.swift @@ -26,7 +26,7 @@ import XCTest /// DO NOT EDIT: Generated GreetMeGreeter messages extension GreetMeGreeter { - public enum Message: ActorMessage { + public enum Message: Codable { case greet(Actor) } diff --git a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/Greeter+GenActor.swift b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/Greeter+GenActor.swift index 1e19a34bb..803d49dd7 100644 --- a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/Greeter+GenActor.swift +++ b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/Greeter+GenActor.swift @@ -31,7 +31,7 @@ import XCTest /// DO NOT EDIT: Generated Greeter messages extension Greeter { - public enum Message: ActorMessage { + public enum Message: Codable { case greet(name: String, _replyTo: ActorRef) } diff --git a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/InvokeFuncs+GenActor.swift b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/InvokeFuncs+GenActor.swift index 866c3103c..e0ca076be 100644 --- a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/InvokeFuncs+GenActor.swift +++ b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/InvokeFuncs+GenActor.swift @@ -31,7 +31,7 @@ import XCTest /// DO NOT EDIT: Generated InvokeFuncs messages extension InvokeFuncs { - public enum Message: ActorMessage { + public enum Message: Codable { case doThingsAndRunTask(_replyTo: ActorRef) case doThingsAsync(_replyTo: ActorRef>) case internalTask(_replyTo: ActorRef) diff --git a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/LifecycleReacting+GenActor.swift b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/LifecycleReacting+GenActor.swift index 14e029ecf..c046dcb36 100644 --- a/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/LifecycleReacting+GenActor.swift +++ b/Tests/DistributedActorsDocumentationTests/Actorable/GenActors/LifecycleReacting+GenActor.swift @@ -31,7 +31,7 @@ import XCTest /// DO NOT EDIT: Generated LifecycleReacting messages extension LifecycleReacting { - public enum Message: ActorMessage { + public enum Message: Codable { case something } diff --git a/Tests/DistributedActorsDocumentationTests/DeathWatchDocExamples.swift b/Tests/DistributedActorsDocumentationTests/DeathWatchDocExamples.swift index add4e4bde..c20333896 100644 --- a/Tests/DistributedActorsDocumentationTests/DeathWatchDocExamples.swift +++ b/Tests/DistributedActorsDocumentationTests/DeathWatchDocExamples.swift @@ -23,7 +23,7 @@ struct Player { } struct GameUnit { - enum Command: ActorMessage { + enum Command: Codable { case player(ActorRef) case otherCommand } @@ -64,7 +64,7 @@ extension GameUnit.Command { } struct GameMatch { - enum Command: ActorMessage { + enum Command: Codable { case playerConnected(ActorRef) case disconnectedPleaseStop } diff --git a/Tests/DistributedActorsDocumentationTests/SerializationDocExamples.swift b/Tests/DistributedActorsDocumentationTests/SerializationDocExamples.swift index 7f8cc4d65..1b1a8b48b 100644 --- a/Tests/DistributedActorsDocumentationTests/SerializationDocExamples.swift +++ b/Tests/DistributedActorsDocumentationTests/SerializationDocExamples.swift @@ -12,20 +12,74 @@ // //===----------------------------------------------------------------------===// +// tag::serialize_manifest_any[] import DistributedActors +import Foundation import NIO +import NIOFoundationCompat +// end::serialize_manifest_any[] // ==== ---------------------------------------------------------------------------------------------------------------- // MARK: Serialization example - Codable messages // tag::serialization_codable_messages[] -enum ParkingSpotStatus: String, ActorMessage { +enum ParkingSpotStatus: String, Codable { case available case taken } // end::serialization_codable_messages[] +// tag::serialization_codable_manual_enum_assoc[] +struct ParkingTicket: Codable {} + +enum ParkingTicketMessage: Codable { + case issued(ParkingTicket) + case pay(ParkingTicket, amount: Int) +} + +extension ParkingTicketMessage { + enum DiscriminatorKeys: String, Codable { + case issued + case pay + } + + enum CodingKeys: CodingKey { // or Int + case _case + case issued_ticket + case pay_ticket + case pay_amount + } + + func encode(to encoder: Encoder) throws { + var container = encoder.container(keyedBy: CodingKeys.self) + switch self { + case .issued(let ticket): + try container.encode(DiscriminatorKeys.issued, forKey: ._case) + try container.encode(ticket, forKey: .issued_ticket) + case .pay(let ticket, let amount): + try container.encode(DiscriminatorKeys.pay, forKey: ._case) + try container.encode(ticket, forKey: .pay_ticket) + try container.encode(amount, forKey: .pay_amount) + } + } + + init(from decoder: Decoder) throws { + let container = try decoder.container(keyedBy: CodingKeys.self) + switch try container.decode(DiscriminatorKeys.self, forKey: ._case) { + case .issued: + let ticket = try container.decode(ParkingTicket.self, forKey: .issued_ticket) + self = .issued(ticket) + case .pay: + let ticket = try container.decode(ParkingTicket.self, forKey: .pay_ticket) + let amount = try container.decode(Int.self, forKey: .pay_amount) + self = .pay(ticket, amount: amount) + } + } +} + +// end::serialization_codable_manual_enum_assoc[] + // ==== ---------------------------------------------------------------------------------------------------------------- // MARK: Serialization example - protobuf messages @@ -74,7 +128,7 @@ extension ParkingGarageStatus { // MARK: Serialization example - custom messages // tag::serialization_custom_messages[] -enum CustomlyEncodedMessage: String { +enum CustomlyEncodedMessage: Codable, NonTransportableActorMessage { case available case taken } @@ -89,8 +143,8 @@ class SerializationDocExamples { func prepare_system_codable() throws { // tag::prepare_system_codable[] - let system = ActorSystem("CodableExample") { _ in -// settings.serialization.registerCodable(ParkingSpotStatus.self, underId: 1002) // TODO: simplify this + let system = ActorSystem("CodableExample") { settings in + settings.serialization.register(ParkingSpotStatus.self) } // end::prepare_system_codable[] _ = system // silence not-used warnings @@ -114,8 +168,8 @@ class SerializationDocExamples { func prepare_system_protobuf() throws { // tag::prepare_system_protobuf[] - let system = ActorSystem("ProtobufExample") { _ in -// settings.serialization.registerProtobufRepresentable(for: ParkingGarageStatus.self, underId: 1002) // TODO: simplify this + let system = ActorSystem("ProtobufExample") { settings in + settings.serialization.register(ParkingGarageStatus.self) } // end::prepare_system_protobuf[] _ = system // silence not-used warnings @@ -138,14 +192,36 @@ class SerializationDocExamples { // MARK: Serialized custom messages func prepare_system_custom() throws { -// // tag::prepare_system_custom[] -// let system = ActorSystem("CustomSerializerExample") { settings in -// settings.serialization.registerSerializer(CustomlyEncodedMessage.self, id: 1001) { allocator in -// CustomlyEncodedSerializer(allocator) -// } -// } -// // end::prepare_system_custom[] -// _ = system // silence not-used warnings + // tag::prepare_system_custom[] + let system = ActorSystem("CustomSerializerExample") { settings in + settings.serialization.registerSpecializedSerializer(CustomlyEncodedMessage.self, serializerID: 1001) { allocator in + CustomlyEncodedSerializer(allocator) + } + } + // end::prepare_system_custom[] + _ = system // silence not-used warnings + } + + func serialization_specific_coder() throws { + // tag::serialization_specific_coder[] + let system = ActorSystem("CustomizeCoderExample") { settings in + settings.serialization.register(MyMessage.self, serializerID: .foundationJSON) + } + // end::serialization_specific_coder[] + _ = system // silence not-used warnings + } + + struct MyMessage: Codable {} + struct OtherGenericMessage: Codable {} + func serialization_register_types() throws { + // tag::serialization_register_types[] + let system = ActorSystem("RegisteringTypes") { settings in + // settings.serialization.insecureSerializeNotRegisteredMessages = false (default in RELEASE mode) + settings.serialization.register(MyMessage.self) + settings.serialization.register(OtherGenericMessage.self) + } + // end::serialization_register_types[] + _ = system // silence serialization_register_types-used warnings } // tag::custom_serializer[] @@ -240,9 +316,77 @@ class SerializationDocExamples { func configure_serialize_all() { // tag::configure_serialize_all[] let system = ActorSystem("SerializeAll") { settings in - settings.serialization.allMessages = true + settings.serialization.serializeLocalMessages = true } // end::configure_serialize_all[] _ = system } } + +// tag::serialize_manifest_any[] + +protocol ForSomeReasonNotCodable {} + +struct DistributedAlgorithmExampleEnvelope: Codable { + let payload: Payload + + enum CodingKeys: CodingKey { + case payload + case payloadManifest + } + + init(from decoder: Decoder) throws { + let container = try decoder.container(keyedBy: CodingKeys.self) + + guard let context = decoder.actorSerializationContext else { + throw SerializationError.missingSerializationContext(decoder, Self.self) + } + + let manifest = try container.decode(Serialization.Manifest.self, forKey: .payloadManifest) + let data = try container.decode(Data.self, forKey: .payload) + var bytes = context.serialization.allocator.buffer(capacity: data.count) + bytes.writeBytes(data) + + // Option 1: raw bytes ---------------------------------------------------------------- + let payload = try context.serialization.deserialize(as: Payload.self, from: &bytes, using: manifest) // <1> + + // Option 2: manually coding + manifest ----------------------------------------------- + // let payloadType = try context.serialization.summonType(from: manifest) // <2> + // if let codablePayloadType = payloadType as? Decodable.Type { + // codablePayloadType._decode(from: bytes, using: decoder) // or some other decoder, you have to know -- inspect the manifest to know which to use + // } + // + // Where _decode is is defined as: + // extension Decodable { + // static func _decode(from buffer: inout NIO.ByteBuffer, using decoder: JSONDecoder) throws -> Self { ... } + // } + + self.payload = payload + } + + func encode(to encoder: Encoder) throws { + guard let context = encoder.actorSerializationContext else { + throw SerializationError.missingSerializationContext(encoder, self) + } + + var container = encoder.container(keyedBy: CodingKeys.self) + + // Option 1: raw bytes ---------------------------------------------------------------- + let (manifest, bytes) = try context.serialization.serialize(self.payload) // TODO: mangled name type manifests only work on Swift 5.3 (!) + try container.encode(manifest, forKey: .payloadManifest) + try container.encode(bytes.getData(at: 0, length: bytes.readableBytes)!, forKey: .payload) // !-safe, we know the range from 0-readableBytes is correct + + // Option 2: manually coding + manifest ----------------------------------------------- + // let manifest_2 = try context.serialization.outboundManifest(type(of: payload as Any)) + // try container.encode(manifest_2, forKey: .payloadManifest) + // let bytes_2 = payload._encode(using: encoder) // or some other encoder, ensure it matches your manifest (!) + // container.encode(bytes_2.getData(at: 0, length: bytes_2.readableBytes)!, forKey: .payload) // !-safe, we know the range from 0-readableBytes is correct + // + // Where _encode is defined as: + // extension Encodable { + // func _encode(using encoder: JSONEncoder, allocator: ByteBufferAllocator) throws -> NIO.ByteBuffer { } + // } + } +} + +// end::serialize_manifest_any[] diff --git a/Tests/DistributedActorsTests/BehaviorTests.swift b/Tests/DistributedActorsTests/BehaviorTests.swift index f7d15539d..853e187a6 100644 --- a/Tests/DistributedActorsTests/BehaviorTests.swift +++ b/Tests/DistributedActorsTests/BehaviorTests.swift @@ -1042,7 +1042,7 @@ final class BehaviorTests: ActorSystemTestBase { let eventLoop = self.eventLoopGroup.next() let promise: EventLoopPromise = eventLoop.makePromise() let future = promise.futureResult - let probe = self.testKit.spawnTestProbe(expecting: NotTransportableAnyError.self) + let probe = self.testKit.spawnTestProbe(expecting: NonTransportableAnyError.self) let error = self.testKit.error() let behavior: Behavior = .setup { context in diff --git a/Tests/DistributedActorsTests/Cluster/RemoteMessagingClusteredTests.swift b/Tests/DistributedActorsTests/Cluster/RemoteMessagingClusteredTests.swift index 924164f5f..0c344870b 100644 --- a/Tests/DistributedActorsTests/Cluster/RemoteMessagingClusteredTests.swift +++ b/Tests/DistributedActorsTests/Cluster/RemoteMessagingClusteredTests.swift @@ -21,13 +21,13 @@ final class RemoteMessagingClusteredTests: ClusteredNodesTestBase { // TODO: This will start failing once we implement _mangledTypeName manifests func test_association_shouldStayAliveWhenMessageSerializationFailsOnReceivingSide() throws { let local = self.setUpNode("local") { settings in - settings.serialization.registerCodable(SerializationTestMessage.self) - settings.serialization.registerCodable(EchoTestMessage.self) + settings.serialization.register(SerializationTestMessage.self) + settings.serialization.register(EchoTestMessage.self) } let remote = setUpNode("remote") { settings in // do not register SerializationTestMessage on purpose, we want it to fail when receiving - settings.serialization.registerCodable(EchoTestMessage.self) + settings.serialization.register(EchoTestMessage.self) } let probeOnRemote = self.testKit(remote).spawnTestProbe(expecting: String.self) @@ -64,8 +64,8 @@ final class RemoteMessagingClusteredTests: ClusteredNodesTestBase { func test_association_shouldStayAliveWhenMessageSerializationThrowsOnSendingSide() throws { let (local, remote) = setUpPair { settings in - settings.serialization.registerCodable(SerializationTestMessage.self) - settings.serialization.registerCodable(EchoTestMessage.self) + settings.serialization.register(SerializationTestMessage.self) + settings.serialization.register(EchoTestMessage.self) } let probeOnRemote = self.testKit(remote).spawnTestProbe(expecting: String.self) @@ -92,8 +92,8 @@ final class RemoteMessagingClusteredTests: ClusteredNodesTestBase { func test_association_shouldStayAliveWhenMessageSerializationThrowsOnReceivingSide() throws { let (local, remote) = setUpPair { settings in - settings.serialization.registerCodable(SerializationTestMessage.self) - settings.serialization.registerCodable(EchoTestMessage.self) + settings.serialization.register(SerializationTestMessage.self) + settings.serialization.register(EchoTestMessage.self) } let probeOnRemote = self.testKit(remote).spawnTestProbe(expecting: String.self) @@ -120,8 +120,8 @@ final class RemoteMessagingClusteredTests: ClusteredNodesTestBase { func test_sendingToRefWithAddressWhichIsActuallyLocalAddress_shouldWork() throws { let local = self.setUpNode("local") { settings in - settings.serialization.registerCodable(SerializationTestMessage.self) - settings.serialization.registerCodable(EchoTestMessage.self) + settings.serialization.register(SerializationTestMessage.self) + settings.serialization.register(EchoTestMessage.self) } let testKit = ActorTestKit(local) @@ -143,9 +143,9 @@ final class RemoteMessagingClusteredTests: ClusteredNodesTestBase { func test_remoteActors_echo() throws { let (local, remote) = setUpPair { settings in - settings.serialization.registerCodable(EchoTestMessage.self) - settings.serialization.registerCodable(SerializationTestMessage.self) - settings.serialization.registerCodable(EchoTestMessage.self) + settings.serialization.register(EchoTestMessage.self) + settings.serialization.register(SerializationTestMessage.self) + settings.serialization.register(EchoTestMessage.self) } let probe = self.testKit(local).spawnTestProbe("X", expecting: String.self) @@ -178,9 +178,9 @@ final class RemoteMessagingClusteredTests: ClusteredNodesTestBase { func test_sendingToNonTopLevelRemoteRef_shouldWork() throws { let (local, remote) = setUpPair { settings in - settings.serialization.registerCodable(EchoTestMessage.self) - settings.serialization.registerCodable(SerializationTestMessage.self) - settings.serialization.registerCodable(EchoTestMessage.self) + settings.serialization.register(EchoTestMessage.self) + settings.serialization.register(SerializationTestMessage.self) + settings.serialization.register(EchoTestMessage.self) } let probe = self.testKit(local).spawnTestProbe("X", expecting: String.self) @@ -221,9 +221,9 @@ final class RemoteMessagingClusteredTests: ClusteredNodesTestBase { func test_sendingToRemoteAdaptedRef_shouldWork() throws { let (local, remote) = setUpPair { settings in - settings.serialization.registerCodable(EchoTestMessage.self) - settings.serialization.registerCodable(SerializationTestMessage.self) - settings.serialization.registerCodable(EchoTestMessage.self) + settings.serialization.register(EchoTestMessage.self) + settings.serialization.register(SerializationTestMessage.self) + settings.serialization.register(EchoTestMessage.self) } let probe = self.testKit(local).spawnTestProbe("X", expecting: String.self) @@ -259,8 +259,8 @@ final class RemoteMessagingClusteredTests: ClusteredNodesTestBase { func test_actorRefsThatWereSentAcrossMultipleNodeHops_shouldBeAbleToReceiveMessages() throws { let (local, remote) = setUpPair { settings in - settings.serialization.registerCodable(SerializationTestMessage.self) - settings.serialization.registerCodable(EchoTestMessage.self) + settings.serialization.register(SerializationTestMessage.self) + settings.serialization.register(EchoTestMessage.self) } remote.cluster.join(node: local.cluster.node.node) @@ -269,8 +269,8 @@ final class RemoteMessagingClusteredTests: ClusteredNodesTestBase { let thirdSystem = self.setUpNode("ClusterAssociationTests") { settings in settings.cluster.bindPort = 9119 - settings.serialization.registerCodable(SerializationTestMessage.self) - settings.serialization.registerCodable(EchoTestMessage.self) + settings.serialization.register(SerializationTestMessage.self) + settings.serialization.register(EchoTestMessage.self) } defer { thirdSystem.shutdown().wait() } diff --git a/Tests/DistributedActorsTests/SerializationPoolTests.swift b/Tests/DistributedActorsTests/SerializationPoolTests.swift index 431d9dae1..d37c75a39 100644 --- a/Tests/DistributedActorsTests/SerializationPoolTests.swift +++ b/Tests/DistributedActorsTests/SerializationPoolTests.swift @@ -95,8 +95,8 @@ final class SerializationPoolTests: XCTestCase { override func setUp() { self.system = ActorSystem("SerializationTests") { settings in - settings.serialization.registerCodable(Test1.self) - settings.serialization.registerCodable(Test2.self) + settings.serialization.register(Test1.self) + settings.serialization.register(Test2.self) } self.testKit = ActorTestKit(self.system) self.elg = MultiThreadedEventLoopGroup(numberOfThreads: 1) diff --git a/Tests/DistributedActorsTests/SerializationTests.swift b/Tests/DistributedActorsTests/SerializationTests.swift index 723cc2ab5..0b6f33faf 100644 --- a/Tests/DistributedActorsTests/SerializationTests.swift +++ b/Tests/DistributedActorsTests/SerializationTests.swift @@ -23,11 +23,11 @@ import XCTest class SerializationTests: ActorSystemTestBase { override func setUp() { _ = self.setUpNode(String(describing: type(of: self))) { settings in - settings.serialization.registerCodable(HasReceivesSystemMsgs.self) - settings.serialization.registerCodable(HasStringRef.self) - settings.serialization.registerCodable(HasIntRef.self) - settings.serialization.registerCodable(HasInterestingMessageRef.self) - settings.serialization.registerCodable(CodableTestingError.self) + settings.serialization.register(HasReceivesSystemMsgs.self) + settings.serialization.register(HasStringRef.self) + settings.serialization.register(HasIntRef.self) + settings.serialization.register(HasInterestingMessageRef.self) + settings.serialization.register(CodableTestingError.self) } } @@ -134,7 +134,7 @@ class SerializationTests: ActorSystemTestBase { func test_serialize_actorRef_inMessage_forRemoting() throws { let remoteCapableSystem = ActorSystem("RemoteCapableSystem") { settings in settings.cluster.enabled = true - settings.serialization.registerCodable(HasStringRef.self) + settings.serialization.register(HasStringRef.self) } let testKit = ActorTestKit(remoteCapableSystem) let p = testKit.spawnTestProbe(expecting: String.self) @@ -267,7 +267,7 @@ class SerializationTests: ActorSystemTestBase { func test_verifySerializable_shouldPass_forPreconfiguredSerializableMessages_string() throws { let s2 = ActorSystem("SerializeMessages") { settings in - settings.serialization.allMessages = true + settings.serialization.serializeLocalMessages = true } do { diff --git a/Tests/DistributedActorsTests/SupervisionTests.swift b/Tests/DistributedActorsTests/SupervisionTests.swift index 29efd1474..cb38716aa 100644 --- a/Tests/DistributedActorsTests/SupervisionTests.swift +++ b/Tests/DistributedActorsTests/SupervisionTests.swift @@ -899,7 +899,7 @@ final class SupervisionTests: ActorSystemTestBase { // MARK: Tests for selective failure handlers /// Throws all Errors it receives, EXCEPT `PleaseReplyError` to which it replies to the probe - private func throwerBehavior(probe: ActorTestProbe) -> Behavior { + private func throwerBehavior(probe: ActorTestProbe) -> Behavior { .receiveMessage { errorEnvelope in switch errorEnvelope.failure { case let reply as PleaseReplyError: @@ -914,7 +914,7 @@ final class SupervisionTests: ActorSystemTestBase { func test_supervisor_shouldOnlyHandle_throwsOfSpecifiedErrorType() throws { let p = self.testKit.spawnTestProbe(expecting: PleaseReplyError.self) - let supervisedThrower: ActorRef = try system.spawn( + let supervisedThrower: ActorRef = try system.spawn( "thrower-1", props: .supervision(strategy: .restart(atMost: 10, within: nil), forErrorType: EasilyCatchableError.self), self.throwerBehavior(probe: p) @@ -936,7 +936,7 @@ final class SupervisionTests: ActorSystemTestBase { func test_supervisor_shouldOnlyHandle_anyThrows() throws { let p = self.testKit.spawnTestProbe(expecting: PleaseReplyError.self) - let supervisedThrower: ActorRef = try system.spawn( + let supervisedThrower: ActorRef = try system.spawn( "thrower-2", props: .supervision(strategy: .restart(atMost: 100, within: nil), forAll: .errors), self.throwerBehavior(probe: p) diff --git a/Tests/GenActorsTests/GenCodableTests.swift b/Tests/GenActorsTests/GenCodableTests.swift index d01e7b583..facd385b0 100644 --- a/Tests/GenActorsTests/GenCodableTests.swift +++ b/Tests/GenActorsTests/GenCodableTests.swift @@ -25,9 +25,9 @@ final class GenCodableTests: XCTestCase { override func setUp() { self.system = ActorSystem(String(describing: type(of: self))) { _ in -// settings.serialization.registerCodable(JackOfAllTrades.Message.self) -// settings.serialization.registerCodable(GeneratedActor.Messages.Parking.self) -// settings.serialization.registerCodable(GeneratedActor.Messages.Ticketing.self) +// settings.serialization.register(JackOfAllTrades.Message.self) +// settings.serialization.register(GeneratedActor.Messages.Parking.self) +// settings.serialization.register(GeneratedActor.Messages.Ticketing.self) } self.testKit = ActorTestKit(self.system) }