Move ByteString to its own independent lib

[pjfanning]

https://contributors.scala-lang.org/t/adding-akkas-bytestring-as-a-scala-module-and-or-stdlib/5967 never really went anywhere but in Pekko 2.0.0, we could refactor so that ByteString and ByteIterator end up in a dedicated lib without other Pekko dependencies. This lib could be used by anyone. The pattern seems generally useful.

I would suggest pekko-bytestring as the lib name.

I would argue that the new moved classes live in a package like org.apache.pekko.bytestring. It is tidier not to split packages like the org.apache.pekko.util package across jars.
We could still keep the pekko-actor ByteString classes but have them extend the new lib classes to avoid code duplication. This means we won't create a source compatibility issue. Or we could add the classes to pekko-actor as a shaded copy of the pekko-bytestring classes. I don't like this latter solution myself but it is an option. I prefer that pekko-actor lib just ends up with a dependency on pekko-bytestring lib.

[He-Pin]

I would.like.+1 for this.

[pjfanning]

It turns out that ByteString supports JDK Serialization. The serialized data includes the fully qualified class name.

Why ByteString Supports JDK Serialization?
ByteString supports JDK serialization primarily because it's used as a message payload in Pekko's actor system, where messages need to be serializable for:

Remote actors – messages sent over the network between actor systems
Pekko Persistence – persisting events/snapshots to a journal
Pekko Cluster – transferring state between nodes

We have no test coverage that ensures that ByteStrings saved using Pekko 1 are readable in Pekko 2 code. There are possibilities that the changes that we already have in main branch don't affect the serialization format.

I'm a little surprised that we haven't already run into issues with people running mixed Akka and Pekko clusters. I don't think too many have done this. We have had people contacting us about this and we have made changes to support it but ByteString never came up.

One option would be to keep the package as is and to rename the package in pekko-actor so that classes that we don't move to pekko-bytestring would use a new package name. I still prefer the idea of not splitting a package across multiple jars (which is discouraged in Java, especially when JPMS is used). Most classes in org.apache.pekko.util are package private but there will probably be something that crops up where one of the classes is public and that we can compat issues.

[He-Pin]

@mdedetrich wdyt about this, not sure if it's possible to land this in Scala 3.9

[He-Pin]

We cannot change the package name; furthermore, we must ensure compatibility between serialization and deserialization to prevent users' persisted data from being unreadable.

[pjfanning]

At this stage, I'm happy enough that this is not all that useful and would be disruptive.

[mdedetrich]

So here are my thoughts.

There is already a project at https://github.com/mdedetrich/bytestring which contains the Pekko bytestring implementation extracted as its own package. I noticed that a decent amount of improvements have been landed in Pekko's bytestring so we would have to cherry pick those over. I didn't end up finishing this project, but that was largely due to bikeshedding as a result of its original implementation not being ideal/perfect but I think there is a good argument that this shouldn't be a deal breaker (in any case, if the abstractions/visibility are done properly nothing is preventing us from changing the underlying implementation to something more performant later on).

If we end up going with this or just renaming the package as is in Pekko, the package name changing isn't a deal breaker, in Pekko 2.0.x we can allow changes to package names and at least for Scala we can provide a type alias that goes from pekko.util.ByteString to scala.collection.immutable.ByteString or w/e it ends up being, unfortunately Java doesn't provide type aliases that would allow a cleanly do this (we could subclass ByteString so its source compatible but that would also mean that ByteString itself has to be non final, but if users feel so strongly about this we can make the initial release of scala.collection.immutable.ByteString 1.0.0 be non-final for this usecase and then change it to final in a later release once people have migrated).

What is definitely a hard line requirement is making sure that the serialization from org.apache.pekko.util.ByteString works with scala.collection.immutable.ByteString.

If we are open to it, I can resume driving the https://contributors.scala-lang.org/t/adding-akkas-bytestring-as-a-scala-module-and-or-stdlib/5967 project and there is a decent change that it can be done before a final release of Pekko (Scala community was quite receptive to it). Otherwise we can move ByteString to its own independant lib while still retaining the org.apache.pekko.util.ByteString but I don't know what that is going to achieve, in my mind we either don't bother at all (at least for Pekko 2.0.x) or we put it in its own lib as a community/sanctified package.

wdyt @He-Pin @pjfanning ?

[He-Pin]

+1 for getting it in stdlib, but it needs to be done in Scala 3.9 LTS

[mdedetrich]

Whats the reason behind Scala 3.9 LTS?

[He-Pin]

So, do you plan to add it to Scala 3.10?