[Feature] Have a flatter directory structure similar to Mill with less boilerplate

[carlosedp]

It would be great to have a flatter directory structure, similar to Mill also in a way that the test module would be nested into it's main module with less boilerplate in the yaml.

Currently the closest using source-layout: normal which is the default is:

❯ tree
./
├── httpserver/
│   ├── src/
│   │   └── scala/
│   │       └── Server.scala
│   └── test/
│       └── src/
│           └── scala/
│               └── ServerSpec.scala
└── bleep.yaml

This requires setting the folders as:

$schema: https://raw.githubusercontent.com/oyvindberg/bleep/master/schema.json
$version: 0.0.2
jvm:
  name: graalvm-java17:22.3.2
projects:
  httpserver:
    dependencies:
      - dev.zio::zio-http:3.0.0-RC2
      - dev.zio::zio:2.0.15
    extends: template-common
    folder: ./httpserver
  tests:
    dependencies:
      - dev.zio::zio-test-sbt:2.0.15
      - dev.zio::zio-test:2.0.15
    dependsOn: httpserver
    extends: template-common
    folder: ./httpserver/test
    isTestProject: true
    testFrameworks: zio.test.sbt.ZTestFramework
templates:
  template-common:
    platform:
      name: jvm
    scala:
      version: 3.3.0

It would be great to have a layout like:

❯ tree
./
├── httpserver/
│   ├── src/
│   │   └── Server.scala
│   └── test/
│       └── src/
│           └── ServerSpec.scala
└── bleep.yaml

Where the isTestProject and the dependsOn settings could be leveraged nesting the test module into httpserver.

$schema: https://raw.githubusercontent.com/oyvindberg/bleep/master/schema.json
$version: 0.0.2
jvm:
  name: graalvm-java17:22.3.2
projects:
  httpserver:
    dependencies:
      - dev.zio::zio-http:3.0.0-RC2
      - dev.zio::zio:2.0.15
    extends: template-common
    tests:
      dependencies:
        - dev.zio::zio-test-sbt:2.0.15
        - dev.zio::zio-test:2.0.15
      isTestProject: true
      testFrameworks: zio.test.sbt.ZTestFramework
templates:
  template-common:
    source-layout: flat
    platform:
      name: jvm
    scala:
      version: 3.3.0

What do you think? Maybe not nesting test leads to less API breakage, so another option would be having the test project in the same level but having a flag indicating to which project it belongs.

[oyvindberg]

Hey @carlosedp , thanks for the input.

You can have a pretty flat structure for the projects if you want! The mechanism works like this:

1. For each given project, bleep first decides the folder, as you have set in your example. this is relative to the build directory.
2. Then bleep looks at source-layout to determine a set of base relative folders where it looks for sources. these are relative to folder from 1). Not that it matters much having a few folders extra here, but this can be set to none for completely customized build layouts.
3. Finally you can add source and resource directories with sources and resources.

If you then say something like this :

  template-common:
    # ...
    source-layout: none
    sources: ./src2

and create directories:

$ bleep build create-directories
📗 (0 ms) Launching Bleep version 0.0.2 as requested in /Users/oyvind/bleep/flaff/bleep.yaml
📙 (5 ms) Refreshing /Users/oyvind/bleep/flaff/.bleep/builds/normal/.bloop...
📗 (11 ms) wrote bloop files [value => Changed: 3]
📗 (11 ms) bootstrapped in 12 ms
📗 (12 ms) Created /Users/oyvind/bleep/flaff/tests/src2 [value => [tests]]
📗 (12 ms) Created /Users/oyvind/bleep/flaff/flaff/src2 [value => [flaff]]

Nested test projects

That's a pass, I'm sorry. These are the reasons:

• the definition of the project and the test project have suprisingly few things in common. you really just save yourself a dependsOn, given that the rest is deduplicated into templates
• in bleep, a core design decision is that project is a project. the production code, the tests, the scripts, the devops, the web site. Just normal projects with dependencies. I know isTestProject muddles the water somewhat, but we've gotta be pragmatic as well :)
• the nesting would imply some amount of similarity which doesn't have to be true. for instance, I sometimes cross-compile my projects but not the tests. That would be extremely surprising with the nesting
• test projects can have dependencies on test projects too - how would would you express that? we would need to come up with some fake names instead of just using exactly what's given in the YAML.
• I don't think there should be one test project per production project, and don't want to encourage it. many builds need exactly one test project in my opinion.

[carlosedp]

Thanks a lot for your amazingly complete answer Oyvind!
That's a great solution using source-layout and sources/resources.

Agree with you on nesting the test projects... sometimes we are so used to do it and don't ask ourselves if it really helps or it's just something we do because the tooling does that way.

The solution is perfectly clean as it is now. I understand the isTestProject param is mostly to identify what is a test project to be able to bleep test and to look for the test framework, right?

Again, thanks a lot for the great tool, I look forward to start using it for some projects!

[carlosedp]

Just for documenting, my sample project became:

# yaml-language-server: $schema=https://raw.githubusercontent.com/oyvindberg/bleep/master/schema.json
$schema: https://raw.githubusercontent.com/oyvindberg/bleep/master/schema.json
$version: 0.0.2
jvm:
  name: graalvm-java17:22.3.2
projects:
  httpserver:
    dependencies:
      - dev.zio::zio-http:3.0.0-RC2
      - dev.zio::zio:2.0.15
    extends: template-common
    platform:
      mainClass: ZioHttpApp
  httpserver-test:
    dependencies:
      - dev.zio::zio-test-sbt:2.0.15
      - dev.zio::zio-test:2.0.15
    dependsOn: httpserver
    extends: template-common
    isTestProject: true
    testFrameworks: zio.test.sbt.ZTestFramework
templates:
  template-common:
    source-layout: none
    sources: ./src
    resources: ./resources
    platform:
      name: jvm
    scala:
      options: -Wunused:all -Wvalue-discard -deprecation -feature -unchecked
      version: 3.3.0

[carlosedp]

BTW, to make the Json schema work on VSCode using the YAML extension (https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml), you need to either configure VSCode settings itself to load the schema or add a modeline to each file.

Globally adding to VSCode settings.json config:

...
    "yaml.schemas": {
        "https://raw.githubusercontent.com/oyvindberg/bleep/master/schema.json": [
            "bleep.yaml"
        ]
    },

...

Or add to each bleep.yaml:

# yaml-language-server: $schema=https://raw.githubusercontent.com/oyvindberg/bleep/master/schema.json

[oyvindberg]

BTW, to make the Json schema work on VSCode using the YAML extension (https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml), you need to either configure VSCode settings itself to load the schema or add a modeline to each file.

Globally adding to VSCode settings.json config:

...
    "yaml.schemas": {
        "https://raw.githubusercontent.com/oyvindberg/bleep/master/schema.json": [
            "bleep.yaml"
        ]
    },

...

Or add to each bleep.yaml:

# yaml-language-server: $schema=https://raw.githubusercontent.com/oyvindberg/bleep/master/schema.json

Thanks for the info! The $schema syntax is finally supported OOTB in intellij now, and I'm hoping vscode converges on the same. if not, maybe bleep setup-ide could setup vscode settings.json in this way - that would be very neat

[carlosedp]

Ah yes, it could add the settings above to the project's .vscode/settings.json file.

[carlosedp]

FYI, I've opened an issue to the YAML Extension...
redhat-developer/vscode-yaml#958