[SPARK-41690][SQL][CONNECT] Agnostic Encoders

[hvanhovell]

What changes were proposed in this pull request?

This PR introduces so AgnosticEncoders. AgnosticEncoders describe how an external type maps to a Spark data type. They are agnostic in the sense that they do not prescribe which internal format is to be used.

For example the following class:

case class Person(id: Long, name: String, hobbies: Seq[String])

Translates into the following agnostic encoder:

ProductEncoder(Person,List(
  (id, PrimitiveLongEncoder),
  (name, StringEncoder),
  (hobbies, IterableEncoder(scala.collection.Seq,StringEncoder))))

This PR integrates AgnosticEncoders in ScalaReflection, and so it is used for all Dataset operations. In a follow-up we will address RowEncoder and JavaReflection. In the old situation we would traverse the type hierarchy for each options (serializedForType, deserializerForType & schemaFor). In the new situation we will create an AgnosticEncoder first and then generate a serializer, deserializer, and/or schema. This saves significantly in time, especially for ExpressionEncoder where we only need one pass through the type hierarchy instead 2 or 3.

Why are the changes needed?

For the Spark Connect Scala Client we need encoders. We want to stay as close to the current Dataset APIs and encoders are part of this. Additionally we would like retain the rich type support.

Encoders are currently tied to ExpressionEncoders, we cannot use for a couple of reasons:

1. Mid-term we don't want to have a dependency on Catalyst. Splitting of the public API that will be shared between Catalyst and the client is tracked in SPARK-41400.
2. ExpressionEncoders only support the internal row format. The client will use Arrow instead.
3. We are nog particularly keen on sending the expressions needed by ExpressionEncoders over the wire. They are overpowered.

So we need an alternative to the current ExpressionEncoders. This class need to be serializable.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing tests.

[hvanhovell]

This is a side effect of not having the same type information when creating the deserializer.

[cloud-fan]

aren't these handled by case _ if isNativeEncoder(enc) => already?

[cloud-fan]

nvm, we have PrimitiveBooleanEncoder and BooleanEncoder

[cloud-fan]

maybe BoxedBooleanEncoder is a better name

[cloud-fan]

thanks, merging to master!