The README.md documents the need for --swift_opt=Visibility=Public as:
The --swift_opt flag passes an option to protoc-gen-swift that makes message classes public, which is necessary to use them with gRPC.
But that explication doesn't make sense. If the .pb.swift files and grpc files are compiled in the same module, then the default internal visibility would be fine.
The real reason one might need public visibility is if the .pb.swift files (and grpc ones) will be placed in a different module from the code that uses them.