Enable referencing proto files in NuGet packages #183
Comments
alexvanboxel
changed the title
|
Your Java example, is it like a csproj PackageReference + Protobuf reference of the contents rolled into one? |
|
I think you could do this today:
|
|
Considerations:
|
|
@JamesNK Could you provide an example? I am trying using my protos from other project in my solution (the idea is latter use nuget), but I can not get it to work. The idea I think is simple, I have a protos project with the proto files in a folder called "Protos". The files are marked as Content.
Then I add that project as reference to mi server project, i can see the references of the files
And I added the Protobuf line But the greet.proto is not detected, what am I missing? |
|
Technically you could make a targets file which enumerated the nuget references and included any .proto files in a well known folder within the packages i.e. 'proto'. might actually be pretty useful if the grpc package did this |
|
Import proto files from nuget packages work under this csproj settings with grpc. csproj proto However, I could not workout how to get the version of the nuget package that I want. Instead, I have to hard code in it. Any thoughts? |
|
This should do everything you want: This will only work with PackageReference (the new format) but i think we should just accept this as a limitation as the old packages.config way is deprecated anyway and would be a lot harder to implement. Output: I'd suggest changing the importance of the first 2 messages printing the search paths to low once you are done testing but leave the others at normal. Let me know if you need any more help, I've done a lot of MSBuild magic lately |
|
Actually it might be worth renaming |
|
Thanks @stueeey , that helps a lot. However, I am having another issue which I couldn't reference the protobuf type in C# code, error says the type (proto message type) could not be found. csproj: proto: Build succeeded, however in C# code: If I removed the target, csproj: Build succeeded and in C# code: Seems like the target element trigger msbuild to do something different, not exactly sure what happen behind the scene of msbuild. But this simple test proof that having under target block does not work for grpc. |
|
@stueeey Did you put that snippet into the csproj of the project that contains the proto file or the one that wants to consume it? I actually need what @JamesYangLim was doing with the AdditionalImportDirs so you can reference the proto file from the nuget package in a proto file in your project. Your snippet does not solve that, does it? |
|
Using GeneratePathProperty on the nuget packge
and adding that path as an protobuf include
seems to solve the problem here? |
|
Hi @llawall, thank you for helping. Your approach works very well and very simple. I have the following example that works. |
|
I had some time today and I've managed to get this working with no changes to the nuget references with the following: This will pick up anything under Works like a dream:
Planning to add a property to the properties page of proto files to make it automatically include the proto in this folder structure when packing the project I'll make a PR into the main GRPC repo - this probably belongs in the GRPC.Tools nuget package |
|
If you want to ship .proto files in a nuget package for reuse in .proto files in your own project you can do this: The key here are the In the MyCompany.ProtoBuf package you need to have something like this: to get the proto files inlcuded. |
|
I wasn't able to get the <ItemGroup>
<Protobuf Include="$(PkgMyCompany_ProtoBuf)\Content\Proto\*.proto" GrpcServices="server" CompileOutputs="true" />
</ItemGroup>
<ItemGroup>
<Protobuf Include="gRPC\**\*.proto" GrpcServices="server" CompileOutputs="true" />
</ItemGroup> |
|
@dameeks: What does this give you? Don't you get the generated client/server already by just referencing the package. Why do you need to generate it again in the consuming project? |
|
@roederja2 My NuGet package contains a single proto file. Adding a reference to the NuGet package adds the proto file to my csproj, but it does build a gRPC client/server. The only way I got it to build was to add a second ItemGroup. I also tried using <Protobuf update="$(PkgMyCompany_ProtoBuf)\Content\Proto\*.proto" >within the same |
by @JamesYangLim works fine for me. |
|
This is very similar to an issue I'm trying to solve. I can't use protos imported into other protos that are linked through a class library. If I use the AdditionalImportDirs flag as seen it will compile, but then when you actually try to reference the something.proto messages in the org.service.proto messages then it says file not found. |
|
Reading the intro more exact would have saved us a lot of time. Let me also stress that this is distributing the proto files and not client libraries. You want to avoid distribution of clients that are build with an older version of proto as the one you are using in your project. @roederja2 ´s advice for shipping protos works perfectly. |
|
@RichyP7 I've done exactly this, they didn't approve it :/ |
|
Thought I'd weigh in as I'm trying to do exactly this: package proto's in nuget package for redistribution such that models can be used in other projects requiring the proto definitions. Specifically sharing 'model' definitions that will be used to build other services. I can package and distribute but inclusion is simply not working using the above methods. dotnet build will setup content links - I can even see them relative to local proto files in the IDE - but will fail with missing files. I can include packaged protos locally only if I manually setup a symlink to the packaged proto files. Maybe this is specific to Linux (I don't have a window machine to test on) and therefore a separate issue altogether. |
|
I have a solution that I can live with for now but it is by no means ideal. It also assumes running in a *nix env which is fine for my needs. Maybe it can help another soul hitting the same issue (MSbuild links are not honoured when compiling local protos using imports from pkg) |
|
@neilwashere if that is all you want to do you can just do this: Includes are cumulative so you can leave your existing Protobuf include where it is if you have local proto files too. My PR which hasn't gone in would allow you to just put your proto files in /protobuf/grpc in your nuget package and they would be automatically found and included without any MSBuild witchery |
I really think this would be an huge improvement but maybe it doesn´t fit the current modular .net approach. Maybe a plugin for nuget would be the right approach like in java. |
|
@stueeey thanks for the reply. Unfortunately in my case, where I need to reference the raw packaged protos in order for my local ones to build, MSBuild just says no. Without an explicit symlink the compiler errors with: |
|
@neilwashere I can probably give you some msbuild magic for that when I'm home |
|
How can it be done when referencing directly another project that contains the proto file? And then I have this in the project that is referencing the project containing the file: Still, seems to not get detected. |
|
This is how I do it. I create a client NuGet package per client. This is what goes in the Client project: <ItemGroup>
<Protobuf Include="..\Server.Project\Protos\*.proto" GrpcServices="Client" AdditionalImportDirs="..\Server.Project\" />
</ItemGroup> |
Ended up with that, thanks! |
|
I just found this discussion. I wish I would have found this discussion sooner. We just spent the last few weeks implementing the exact same solution which was to share protos as content via a NuGet package. My team has mixed feelings on this. The concern they have is that we are checking in the protos twice or more. Once in the main repo for the protos and then again for any consuming project. Personally, I am not worried about this. There are bigger problems to solve, hence my next point. Where this falls down is when you want to share common protos across multiple services. This wouldn't be an issue if a consumer only talked to one service. We have situations where one consumer can talk to multiple services. After all, we are in a world of microservices. Also, we do not cross the streams. Our clients are generated in separate NuGet packages and by extension, separate assemblies. One per service! Some languages this may not be a problem, but in .NET, it is a big deal. The challenge is that common types generated from the common protos are seen in two different assemblies. This should be doable. After all, we do not have to have the protos for the wellknown protos (ex. timestamp) from Google. Those are determined by the gRPC tooling and you are allowed to reference them. Why can't we do the same with our own wellknown (i.e. common protos)? If I am missing any information, please educate me. Reference to the issue I just created as well as my current path to solving this problem which is different, but similar. |
|
Hey @JamesNK @jtattermusch , what needs to happen here? It seems like there is demand for this, what needs to happen for the pr to be accepted? |
|
Hello, if you need an alternative way to share .proto files and other artefacts using NuGet packages take a look here https://github.com/albanian-xrm/greeter.shared and here https://www.linkedin.com/pulse/source-only-nuget-packages-albanianxrm |
|
I have created a demo project based on @stueeey's comment. To me this is a great way to solve the problem, but unfortunately Visual Studio 2022 and Visual Studio Code (OmniSharp) will display red squiggles in the editor leading to a very bad editing experience. I have provided feedback to Visual Studio but unfortunately I seem to be the only one suffering from this. |
|
@Liversage have you tried my way in the previous comment? |
|
@BetimBeja No, the shared project ( |
|
You actually don't need to use the Shared Project... you can configure the .targets and .props with the necessary "imports" and they will get automatically included in the consuming project. |
I have created a solution/work around that fixes the red squiggles in the editor. It adds a bit of friction to using |
|
@JamesNK we are trying to use the nuget packaging approach for building Rest APIs with the gRPC HTTP API concept. When referencing a nuget package with generated server code the application both builds and runs properly, but the services are not recognized and generated as controllers, swagger also cant find the server-code from nuget. We use a shared "contracts" mono-repo for the .proto files and code related to generation. Is there a good example of this workflow? |
This was helpful although the build failed due to an import statement in one of the proto files . The below fix the issue for me: |
|
Looks like @JamesNK have figured out how to properly make nugets with proto-files here - dotnet/aspnetcore#44999. |
Thanks! This below works for our specific project. |
|
Does this solution also works with |
|
This requires nuget publish. Does |
thanks. I've ended up referencing protos by path in the same solution. |
…jects and nuget packages (#34521) Added documented for Grpc.Tools for sharing proto files - between projects in a solution - in NuGet packages to be used by other projects This addresses the questions raised in: - grpc/grpc-dotnet#1458 - grpc/grpc-dotnet#183 @jtattermusch @JamesNK --------- Co-authored-by: James Newton-King <james@newtonking.com>
|
Grpc.Tools BUILD-INTEGRATION.md has now been updated with how to share proto files. |
and others
A common pattern in other languages is that your proto files are not part of your project, especially if you are a client but ofter also when you are a server.
Certainly in enterprises that have their stack build on gRPC. What you see is that the proto's are managed in a common mono repo for all contracts (were talking mono-repo for the proto files only here). What you then often see is that contract repo has a language specific distribution step. This do not create client libraries, they just publish the proto files in a package format idiomatic to the language (an example is Java: they package the proto files in a jar package and publish them to a maven repo, in general a private one of the company).
What I like to see is that the .NET implementation would be able to reference proto in NuGet that it can fetch from a NuGet repository. That will make it easier for enterprises to adapt gRPC in .NET. They only need to add the NuGet distribution step in one place and a project just need to references NuGet as an external dependency.
Let me also stress that this is distributing the proto files and not client libraries. You want to avoid distribution of clients that are build with an older version of proto as the one you are using in your project.
And example how this works in Java (using gradle):
The 3th dependency points to an artifact just containing proto, gradle will make sure that client/server stubs are generate with the same version of gRPC and protobuf as the project is.
The text was updated successfully, but these errors were encountered: