Support Protobuf Imports

[CEikermann]

Fixes #237

Fixed by removing the "/" at the beginning of all file paths and added "/" as import path to the proto parser

[weeco]

Can you explain why the default (currentDirectory) was an issue and / (root directory) works apparently? I'm unsure if I understand the actual circumstances causing the issue.

[CEikermann]

Without the removal of the "/" at the beginning of the filePath, the parser is failing.

Error Log:

{"level":"warn","ts":"2021-09-08T16:05:25.599Z","msg":"failed to parse proto file to descriptor","file":"test_b.proto","line":5,"error":"test_b.proto:5:1: duplicate symbol test.b.protobuf.TestB: already defined as message in \"/test_b.proto\""}
{"level":"warn","ts":"2021-09-08T16:05:25.610Z","msg":"failed to parse proto file to descriptor","file":"test_b.proto","line":6,"error":"test_b.proto:6:3: duplicate symbol test.b.protobuf.TestB.test: already defined as field in \"/test_b.proto\""}
{"level":"fatal","ts":"2021-09-08T16:05:25.611Z","msg":"failed to start kafka service","error":"failed to create proto registry: failed to compile proto files to descriptors: failed to parse proto files to descriptors: parse failed: invalid proto source"}

My guess the files are parsed twice, once via input "/test_b.proto" and once via import "test_b.proto"
When we are removing the "/" at the beginning, seems the proto parser detects that the same file has been parsed already and skip it.

[weeco]

Thanks for taking a look and finding this!

[CEikermann]

Thanks for merging! :)

[weeco]

Just FYI @CEikermann this broke protobuf support if the Git provider (instead of file provider) is used. Can you post what your protobuf configuration looked like, so that I can figure out a solution that works for both?

[CEikermann]

Just FYI @CEikermann this broke protobuf support if the Git provider (instead of file provider) is used. Can you post what your protobuf configuration looked like, so that I can figure out a solution that works for both?

Sorry this was not intended.

#237 (comment) is this enough for you to reproduce? You can just use version 1.4 to reproduce this bug

[weeco]

@CEikermann I just tested this locally and I was not able to reproduce this. It might be because I'm running on windows at the moment, but I could probably also test on Mac. I tend to revert your changes made here because of that.

In my case an import like address.proto ends up in ddress.proto and that's of course not existent in my in memory filesystem. Hence just removing the first character can not be the solution after all :/

[weeco]

I just pushed a version that removes the prefix / on all filepaths. Maybe that helps with the filepath mismatch. Could you try that? Otherwise I'd like to figure out how to reproduce your issue so that I can resolve them in a more methodical way.

[CEikermann]

@weeco I have tested with master 11afdfa

I do get an error when launching the kowl docker container

{"level":"fatal","ts":"2021-09-13T18:54:17.086Z","msg":"failed to start kafka service","error":"failed to create proto registry: failed to compile proto files to descriptors: failed to parse proto files to descriptors: test_b.proto: file does not exist"}

[CEikermann]

As addition to #237 (comment)

docker-compose.yml

  kafka-ui:
    image: quay.io/cloudhut/kowl:master-11afdfa
    restart: unless-stopped
    volumes:
      - ./docker/kafka-ui/config.yml:/etc/kowl/config.yml
      - /tmp/protobuf/:/etc/protos/
    ports:
      - 9093:8080
    entrypoint: ./kowl --config.filepath=/etc/kowl/config.yml
    depends_on:
      - kafka
    networks:
      - main

Docker Desktop 4.0.0 @ MacOS 11.4

You can not reproduce this on your env with the given files?

[CEikermann]

@weeco are you able to reproduce the issue on your env?

[weeco]

@CEikermann Yes, I can reproduce it using Docker, not within my IDE unfortunately. So I'm not entirely sure what's going on there, but I will try to figure it out as soon as I have some time for it

[CEikermann]

Have created an follow up issue of this: #265