fix: dont commit generated files to git

[cryptk]

Description

Generated files should not be committed to version control, only the files that they are generated from. Generated files (especially protobuf/grpc files) cause a large amount of noise in linters and other IDE extensions as they typically don't conform to anything even approaching a standard style and frequently reference things that are not defined until build time.

This commit removes the protobuf generated files and improves the Makefile to generate these files at build time. This has the added benefit of ensuring that the project is always built with the latest version of the protobuf spec.

There are also some documentation improvements to help new users to ensure they have the tooling required to compile the relevant protobuf files.

Notes for Reviewers

Signed commits

• Yes, I signed my commits.

[netlify]

✅ Deploy Preview for localai canceled.

Name	Link
🔨 Latest commit	4bab32d
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/66197496793089000845312c

[mudler]

I think this overall is the direction we should take indeed, thanks for having a look at this

There are also some documentation improvements to help new users to ensure they have the tooling required to compile the relevant protobuf files.

This was the strongest point that demotivated me to switch to in-flight generation of protobuf file to begin with - since few users (especially Apple) would need to build from source in some cases, the amount of tooling needed would be likely a barrier. But I think we should just go ahead with this and try to tackle that separately (we could be more consistent and offer pre-built binaries as now Github offers Apple arm64 workers! actions/runner-images#9254 )

[cryptk]

I think this overall is the direction we should take indeed, thanks for having a look at this

There are also some documentation improvements to help new users to ensure they have the tooling required to compile the relevant protobuf files.

This was the strongest point that demotivated me to switch to in-flight generation of protobuf file to begin with - since few users (especially Apple) would need to build from source in some cases, the amount of tooling needed would be likely a barrier. But I think we should just go ahead with this and try to tackle that separately (we could be more consistent and offer pre-built binaries as now Github offers Apple arm64 workers! actions/runner-images#9254 )

Another idea I had is potentially adding an option to run the build inside a docker container so that you don't have to worry about configuring your environment, instead, there will be an environment configured for you that runs the build and you get the artifact out of it. I think that's a potential future enhancement as well though.

[dave-gray101]

I think this overall is the direction we should take indeed, thanks for having a look at this

There are also some documentation improvements to help new users to ensure they have the tooling required to compile the relevant protobuf files.

This was the strongest point that demotivated me to switch to in-flight generation of protobuf file to begin with - since few users (especially Apple) would need to build from source in some cases, the amount of tooling needed would be likely a barrier. But I think we should just go ahead with this and try to tackle that separately (we could be more consistent and offer pre-built binaries as now Github offers Apple arm64 workers! actions/runner-images#9254 )

I'll verify this next time I'm at my build mac, but as of the last time I checked, the current homebrew versions of protobuf worked for OSX just fine - I don't think we need to consider OSX users a blocker here, as the basic build prereqs should be sufficient.

[cryptk]

I think this overall is the direction we should take indeed, thanks for having a look at this

There are also some documentation improvements to help new users to ensure they have the tooling required to compile the relevant protobuf files.

This was the strongest point that demotivated me to switch to in-flight generation of protobuf file to begin with - since few users (especially Apple) would need to build from source in some cases, the amount of tooling needed would be likely a barrier. But I think we should just go ahead with this and try to tackle that separately (we could be more consistent and offer pre-built binaries as now Github offers Apple arm64 workers! actions/runner-images#9254 )

I'll verify this next time I'm at my build mac, but as of the last time I checked, the current homebrew versions of protobuf worked for OSX just fine - I don't think we need to consider OSX users a blocker here, as the basic build prereqs should be sufficient.

The only system that I had to do anything special with to get it to build were the Ubuntu 22.04 images. Mostly because 22.04 comes with some very old packages, so I needed to update protoc and I wasn't able to use python gpio tools from the repository either.

The Apple tests are passing, and they were actually one of the easier tests to get working, but if you could run some builds with this change and make sure that everything seems kosher, that would be great!

[golgeek]

I'll verify this next time I'm at my build mac, but as of the last time I checked, the current homebrew versions of protobuf worked for OSX just fine - I don't think we need to consider OSX users a blocker here, as the basic build prereqs should be sufficient.

Agreed, had to make protobuf yesterday on my arm64 mac for #1990 and had no issue.

Basically:

$ brew install protobuf
$ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
$ pip install grpcio-tools

The only thing that didn't go as plan is that I used a conda environment, and somehow python3 is not linked to the conda env, so I had to replace python3 with python in the Makefile. Though this is definitely linked to my setup.

[mudler]

nice! that's a great cleanup, really liking it!