Issues with List type while using zio-config-yaml along with yaml resource files

[ksarath]

Hi,

zio-config-yaml fails to load the configuration properly when there is List type in configuration. We are using Scala 3, ZIO: 1.0.13, zio-config: 2.0.0, zio-config-yaml: 2.0.0 and zio-config-magnolia: 2.0.0.

Here are some examples. The failure happens when we use fromYamlReader() and when we use <> (orElse) operator together with fromYamlFile(). To illustrate this, I am using two resource files application.yml and application-dev.yml

application.yml

kafkaClients:
    security:
        protocol: SSL
    timeoutInSeconds: 300
    bootstrapServers: 
    - localhost:9092
    - localhost:9094

application-dev.yml

kafkaClients:
    security:
        protocol: PLAINTEXT

YamlConfigSpec.scala

import zio.config.magnolia.descriptor
import zio.config.read
import zio.config.yaml.YamlConfigSource
import zio.test.{assertM, Assertion, DefaultRunnableSpec, ZSpec}
import zio.test.Assertion.equalTo

import java.io.File

import scala.io.Source

object YamlConfigSpec extends DefaultRunnableSpec:
  case class KafkaSecurity(protocol: String)
  case class KafkaClients(bootstrapServers: List[String], timeoutInSeconds: Int, security: KafkaSecurity)
  case class ApplicationConfig(kafkaClients: KafkaClients)
  val applicationConfigDescriptor = descriptor[ApplicationConfig]

  case class KafkaClientsWithoutList(timeoutInSeconds: Int, security: KafkaSecurity)
  case class ApplicationConfigWithoutBootstrapServers(kafkaClients: KafkaClientsWithoutList)
  val applicationConfigWithoutListDescriptor = descriptor[ApplicationConfigWithoutBootstrapServers]

  def spec: ZSpec[Environment, Failure] =
    suite("YamlConfigSourceTest")(
      testM("Reading configuration from only application.yml - Passing test") {
        val applicationYamlSource = YamlConfigSource.fromYamlFile(new File("src/test/resources/application.yml"))
        val expected =
          ApplicationConfig(
            KafkaClients(List("localhost:9092", "localhost:9094"), 300, KafkaSecurity("SSL"))
          )
        val result = read(applicationConfigDescriptor from applicationYamlSource)

        assertM(result)(equalTo(expected))
      },
      testM("Reading configuration from application-dev.yml and application.yml - Failing test") {
        val applicationDevYamlSource = YamlConfigSource.fromYamlFile(new File("src/test/resources/application-dev.yml"))
        val applicationYamlSource    = YamlConfigSource.fromYamlFile(new File("src/test/resources/application.yml"))
        val expected =
          ApplicationConfig(
            KafkaClients(List("localhost:9092", "localhost:9094"), 300, KafkaSecurity("PLAINTEXT"))
          )
        val result = read(applicationConfigDescriptor from (applicationDevYamlSource <> applicationYamlSource))

        assertM(result)(equalTo(expected))
      },
      testM("Reading configuration from application-dev.yml and application.yml (without List type) - Passing test") {
        val yamlSource1 = YamlConfigSource.fromYamlFile(new File("src/test/resources/application.yml"))
        val yamlSource2 = YamlConfigSource.fromYamlFile(new File("src/test/resources/application-dev.yml"))
        val expected =
          ApplicationConfigWithoutBootstrapServers(
            KafkaClientsWithoutList(300, KafkaSecurity("PLAINTEXT"))
          )
        val result = read(applicationConfigWithoutListDescriptor from (yamlSource2 <> yamlSource1))

        assertM(result)(equalTo(expected))
      },
      testM("Reading configuration from application yaml using reader - Failing test") {
        val applicationYamlSource =
          YamlConfigSource.fromYamlReader(Source.fromFile("src/test/resources/application.yml").reader)
        val expected =
          ApplicationConfig(
            KafkaClients(List("localhost:9092", "localhost:9094"), 300, KafkaSecurity("SSL"))
          )
        val result = read(applicationConfigDescriptor from applicationYamlSource)

        assertM(result)(equalTo(expected))
      },
      testM("Reading configuration from application yaml using reader (without List type) - Passing test") {
        val applicationYamlSource =
          YamlConfigSource.fromYamlReader(Source.fromFile("src/test/resources/application.yml").reader)
        val expected =
          ApplicationConfigWithoutBootstrapServers(
            KafkaClientsWithoutList(300, KafkaSecurity("SSL"))
          )
        val result = read(applicationConfigWithoutListDescriptor from applicationYamlSource)

        assertM(result)(equalTo(expected))
      }
    )

[ksarath]

[afsalthaj]

@ksarath I will take a look at this one.
Quite busy these days, give me a couple of days.

[afsalthaj]

Sorry for the delay. Spotted the issue with RC versions.
#815
Hopefully, be trying to release soon.

[afsalthaj]

Currently working on testM("Reading configuration from application yaml using reader - Failing test") , but it seems to be a different issue, as it is source specific. They are only failing for Yaml Source, and not for a Hocon Source.

[afsalthaj]

Another issue that I found in this RC version is lazy evaluation of trees even after memoization. This will be fixed as part of the PR.

[afsalthaj]

This is ready for review, with an example inspired from the issue itself:
https://github.com/zio/zio-config/blob/fix/783/examples/shared/src/main/scala/zio/config/examples/ConfigSourceOrElseExample.scala

[afsalthaj]

Also included a reader example, although we are thinking whether to be hesitant about having exposing that in yamlSource. But not a concern of this PR.

https://github.com/zio/zio-config/blob/fix/783/examples/shared/src/main/scala/zio/config/examples/ConfigSourceOrElseExample.scala#L56

[afsalthaj]

Recommending to use https://github.com/zio/zio-config/releases/tag/v2.0.4.
See if it fixes, if not let us know.

[afsalthaj]

Reopening the issue, just to allow users to test the latest RC versions that works with latest zio RC versions.

https://github.com/zio/zio-config/releases/tag/v3.0.0-RC5

(along with corresponding 2.x version: https://github.com/zio/zio-config/releases/tag/v2.0.4)