Lacking support of `protoc` equivalent `--proto_path` option

[zhouyan]

Maybe I missed it, but there seems to be no support for something similar to --proto_path or -I option as found for other languages's protoc based implementation. For example,

import "google/protobuf/timestamp.proto";

will not work out of box. And this is common enough usage of ProtoBuf.

[murraystokely]

This is supported with readProtoFiles. e.g. readProtoFiles('google/protobuf/timestamp.proto') readProtoDir('google/protobuf')

…

On Sat, Jan 6, 2018 at 4:08 PM, Yan Zhou ***@***.***> wrote: Maybe I missed it, but there seems to be no support for something similar to --proto_path or -I option as found for other languages's protoc based implementation. For example, import "google/protobuf/timestamp.proto"; will not work out of box. And this is common enough usage of ProtoBuf. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#39>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEwu7a0r8rxIenokz6F-tcOM4-gj0FfYks5tIAr1gaJpZM4RVfEY> .

[zhouyan]

Thanks for the reply. I meant when these files are included in other proto files, for example, `readProtoFiles("foo.proto")` where `foo.proto` has a import statement that include other files.

[eddelbuettel]

Well

• Maybe collate them by hand
• Work out a PR to add an include mechanism

It so happens that I was musing about an #include equivalent for reading TOML files (in C++) and my pal @dcdillon actually developed a fully generic solution for input streams (which we still need to showcase better). That could work here too, but someone needs to do the work, and you are the one with the itch.

Unless, of course, @murraystokely comes up with another google-y trick already present in the library. I half expect it to be there too...

[zhouyan]

For example,

import "google/protobuf/timestamp.proto";

message Foo {
    google.protobuf.Timestamp ts = 1;
}

readProtoFiles("foot.proto")

will always lead to error google/protobuf/timestamp.proto:0:1:File not found.

A workaround I find is that to put all dependencies in one directory, for example, say the current working directory is as the following,

• google/protobuf/timestamp.proto
• foo.proto

Then it works. But now consider that foot.proto is to be distributed with package Foo, And there's another bar.proto to be distributed with package Bar. If only one of them is used, then we can put google/protobuf along with the package protofiles and change working directory temporarily in some package initialization steps. However, because now timestamp.proto appears in two places, and try to read it twice from two places will leads to error like "google.protobuf.Timestamp" is already defined in file "google/protobuf/timestamp.proto".

What's worse, say now package Foo depends on package Bar and want to import bar.proto, it just get even messier.

In short, the current implementation only really work for self-contained proto files, which isn't really the most commonly usage of ProtoBuf in general

[murraystokely]

This already works you can readProtoFiles(dir='google/protobuf') readProtoFiles(foo.proto) where foo.proto imports google/protobuf/timeseries.proto Your example isn't fully self contained so I can't make it work exactly as written, but yes you can certainly include large libraries of .proto files from different directories and import them in your .protos with earlier versions of RProtoBuf. If you package up a self-contained tarball or something with two .proto files I may be able to help get it working. - Murray

…

On Sat, Jan 6, 2018 at 5:00 PM, Yan Zhou ***@***.***> wrote: For example, import "google/protobuf/timestamp.proto"; message Foo { google.protobuf.Timestamp ts = 1; } readProtoFiles("foot.proto") will always lead to error google/protobuf/timestamp.proto:0:1:File not found. A workaround I find is that to put all dependencies in one directory, for example, say the current working directory is as the following, - google/protobuf/timestamp.proto - foo.proto Then it works. But now consider that foot.proto is to be distributed with package Foo, And there's another bar.proto to be distributed with package Bar. If only one of them is used, then we can put google/protobuf along with the package protofiles and change working directory temporarily in some package initialization steps. However, because now timestamp.proto appears in two places, and try to read it twice from two places will leads to error like "google.protobuf.Timestamp" is already defined in file "google/protobuf/timestamp.proto". What's worse, say now package Foo depends on package Bar and want to import bar.proto, it just get even messier. In short, the current implementation only really work for self-contained proto files, which isn't really the most commonly usage of ProtoBuf in general — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEwu7Rl2JLYWPhVVjn3oYkksg1lrULHlks5tIBcUgaJpZM4RVfEY> .

[siddharthab]

The --proto_path option that you mention is passed on as getwd() and the directory of the included files by default. There is currently no intuitive mechanism for providing a custom directory, other than the indirect way (by making your current dir the same as the include path and using dir instead of files) mentioned by @murraystokely above which is difficult to discover, and will not work when there are two include directories that have imports into each other.

rprotobuf/R/internals.R

Line 46 in 8219122

directories <- unique( c( getwd(), dirname(files) ) )

There is another strange behavior in which files argument are converted to their absolute paths. So if RProtoBuf reads them once as their absolute path, every import statement in a subsequently read proto file should also refer them with their absolute paths to match the originally read file.

I tried to make this behavior consistent with protoc, but it was difficult to do so without introducing breaking changes from existing behavior, especially where file descriptors (and import statements for them) are being keyed by absolute paths.

So I introduced a new function in #40.

[jcaberio]

hello.
i've manage to make a minimal package that demonstrates the need for --proto_path option.

https://github.com/jcaberio/pbimport/tree/master

[siddharthab]

This is now resolved by #40.

[eddelbuettel]

@siddharthab Is there an email address I can use for you in the ChangeLog ?

[siddharthab]

bagaria.siddhartha@gmail.com

Thank you again!

[eddelbuettel]

Lovely, thanks. If you happen to have a unit test or two hanging around for this, or feel like cooking one up I would gladly take them to complete this further. NEWS and ChangeLog entry coming in a minute...

[siddharthab]

I have these two commits -- master...siddharthab:unit

Description of these tests is in #40 (comment)

The tests will work but to make them proper, we should also be clearing persisted state between two unit tests. We will need a utility function in C++ called before/after each test.

[eddelbuettel]

Right, I recalled some tests being part of the discussion.

Is it woth cooking up a quick helper function to purge state? Do you want to give it a shot?

[siddharthab]

Done -- #45