🐞 Cannot load multiple versions of the same module to the same shell session

[rvernica]

What is the issue?

dagger session destroy command is stuck. I can't stop it with Ctrl-C.

Dagger version

dagger v0.18.12 (docker-image://registry.dagger.io/engine:v0.18.12) linux/amd64

Steps to reproduce

Exit dagger shell with exit and run the above command.

Log output

Not sure where the logs are..

[jedevc]

Any specific reason you're using the dagger session command? It's meant to be an internal command, only used by the SDKs to spin up a session. It's not meant to be called directly.

[rvernica]

I was trying to load a different version of the trivy module, and it conflicted with the local one. I was hoping the session destroy and mod cache prune commands would clear the cache, and I could get the intended version of trivy. This is the error I was trying to work around:

! failed to serve module: module trivy (source: github.com/jpadams/daggerverse/trivy@44c178290a412483e785a436aabc46c707842a62 | pin:
44c178290a412483e785a436aabc46c707842a62) already exists with different source

  github.com/jpadams/daggerverse/trivy@1e7077df01ea8c849620c6fe2e7993b223c48bb4 (pin: 1e7077df01ea8c849620c6fe2e7993b223c48bb4)

[jedevc]

dagger session destroy isn't a dagger command. Neither is mod cache prune? I'm confused, where are you getting these commands from? To clear the cache, you can use dagger core engine local-cache prune.

Can you share some more context? How are you getting the error that you're sharing?

[rvernica]

Thanks for looking into this. This is the original issue I was trying to work around:

▶ github.com/jpadams/daggerverse/trivy@44c178290a412483e785a436aabc46c707842a62 | scan-image alpine:latest 34.0s
                                                                                                                     
Report Summary                                                                                                       
                                                                                                                     
┌───────────────────────────────┬────────┬─────────────────┬─────────┐                                               
│            Target             │  Type  │ Vulnerabilities │ Secrets │                                               
├───────────────────────────────┼────────┼─────────────────┼─────────┤                                               
│ alpine:latest (alpine 3.22.0) │ alpine │        0        │    -    │                                               
└───────────────────────────────┴────────┴─────────────────┴─────────┘                                               
Legend:                                                                                                              
- '-': Not scanned                                                                                                   
- '0': Clean (no security findings detected)                                                                         

▼ github.com/jpadams/daggerverse/trivy@v0.5.0 | scan-image alpine:latest 11.5s ERROR
╰─▼ github.com/jpadams/daggerverse/trivy@v0.5.0 11.5s ERROR
  ! failed to serve module: module trivy (source: github.com/jpadams/daggerverse/trivy@trivy/v0.5.0 | pin:
    1e7077df01ea8c849620c6fe2e7993b223c48bb4) already exists with different source
    github.com/jpadams/daggerverse/trivy@44c178290a412483e785a436aabc46c707842a62 (pin:
    44c178290a412483e785a436aabc46c707842a62)
  ├─▶ detect module: github.com/jpadams/daggerverse/trivy@v0.5.0 1.1s
  ╰─▼ load module: github.com/jpadams/daggerverse/trivy@trivy/v0.5.0 10.4s ERROR
    ! failed to serve module: module trivy (source: github.com/jpadams/daggerverse/trivy@trivy/v0.5.0 | pin:
      1e7077df01ea8c849620c6fe2e7993b223c48bb4) already exists with different source
      github.com/jpadams/daggerverse/trivy@44c178290a412483e785a436aabc46c707842a62 (pin:
      44c178290a412483e785a436aabc46c707842a62)
    ├─● finding module configuration 0.9s
    ╰─▼ initializing module 9.7s ERROR
      ! module trivy (source: github.com/jpadams/daggerverse/trivy@trivy/v0.5.0 | pin:
        1e7077df01ea8c849620c6fe2e7993b223c48bb4) already exists with different source
        github.com/jpadams/daggerverse/trivy@44c178290a412483e785a436aabc46c707842a62 (pin:
        44c178290a412483e785a436aabc46c707842a62)
      ├─● ModuleSource.asModule: Module! 9.7s
      ╰─✘ .serve(includeDependencies: true): Void 0.0s ERROR
        ! module trivy (source: github.com/jpadams/daggerverse/trivy@trivy/v0.5.0 | pin:
          1e7077df01ea8c849620c6fe2e7993b223c48bb4) already exists with different source
          github.com/jpadams/daggerverse/trivy@44c178290a412483e785a436aabc46c707842a62 (pin:
          44c178290a412483e785a436aabc46c707842a62)
! failed to serve module: module trivy (source: github.com/jpadams/daggerverse/trivy@trivy/v0.5.0 | pin:
  1e7077df01ea8c849620c6fe2e7993b223c48bb4) already exists with different source
  github.com/jpadams/daggerverse/trivy@44c178290a412483e785a436aabc46c707842a62 (pin:
  44c178290a412483e785a436aabc46c707842a62)

Notice that the first command works while the second one, where I'm trying to use a different version of trivy fails. This is the reason I was trying to clear the module cache.

[jedevc]

Oh interesting.

Looks like we're trying to load two different versions of the module into the same session... which explicitly breaks, because they would have the same name.

@helderco what's the desired behavior here?

[helderco]

@helderco what's the desired behavior here?

The shell supports different versions in terms of introspection and I've used that, but I suppose if the schemas are incompatible it'll fail in the engine so it needs to be solved there. Not sure what's best, to fail or to replace. I'm leaning on the latter, what would be the consequences of that?

[jedevc]

I think replacing makes sense.

But suppose you do:

$ FOO=$(mod-name@v1 | do-thing)
$ mod-name@v2 | do-thing
<output from v2>
$ $FOO
<output from v1>

Are we just kind of constantly loading/unloading modules? It feels a bit annoying to do that (but definitely better than the current behavior)

I wonder if we need to somehow support multiple names being loaded into a session.

[helderco]

Are we just kind of constantly loading/unloading modules?

Not sure what "unloading" would be but from the CLI's POV the introspection from different versions stick around because they're cached by the module source's digest. It's the engine that can't serve incompatible versions in the same session. So it depends on how well the engine can switch serving one vs the other. Maybe both can stay loaded and when calling serve it just chooses which one to "expose"?