Skip to content

gRPC API modules#8

Merged
vigoo merged 35 commits into
mainfrom
grpc_api
Jan 19, 2024
Merged

gRPC API modules#8
vigoo merged 35 commits into
mainfrom
grpc_api

Conversation

@justcoon
Copy link
Copy Markdown
Contributor

@justcoon justcoon commented Jan 1, 2024

grpc api extracted from golem-common and moved/split to golem-api-grpc and golem-api-grpc-cloud

  • OSS common types and structure are in golem.common package, services and related types have own package, for example TemplateService have golem.template package
  • cloud protos are in golem.cloud package, common types and structure are in golem.cloud.common package, services have own package and Cloud prefix, for example CloudProjectService have golem.cloud.project package,
  • there are 2 TemplateService-s (OSS TemplateService and CloudTemplateService), as there are differences in case of ProjectId
  • GolemError was refactored to golem.worker.WorkerExecutionError
  • TokenSecret was removed from request payloads, metadata (headers) will be used for authorisation token propagation
  • golem-cloud-server-oss - grpcapi impl.

TODO

  • golem.worker.WorkerExecutionError variants have naming like InvalidRequest, WorkerAlreadyExists ..., which in protobuf namespace is golem.worker.InvalidRequest, golem.worker.WorkerAlreadyExists, at this point this may easily make naming conflict, they should have WorkerExecutionError in name, probably like: golem.worker.InvalidRequestWorkerExecutionError, golem.worker.WorkerAlreadyExistsWorkerExecutionError
  • golem-api-grpc-cloud - need golem-api-grpc proto files for compilation, we need to figure out how we provide those dependencies when golem-api-grpc-cloud will be moved to cloud projects
  • probably in next PR: move AccountId and ResourceLimits from golem-api-grpc to golem-api-grpc-cloud - now those are used in worker executor OSS impl., remove authorisation from worker executor OSS impl.

@afsalthaj
Copy link
Copy Markdown
Contributor

Firstly, its not scalable idea to reuse services. I hope we don't have common Service types. You mentioned TemplateService is duplicated, and I believe it should be.

@afsalthaj
Copy link
Copy Markdown
Contributor

afsalthaj commented Jan 1, 2024

Also , I believe, if we have dedicated packages already for template , worker etc (and similarly in golem.cloud package, we have project, account etc) their corresponding types should also be in the same package, instead of scattering these types into golem.common (and golem.cloud.common). May not be for this PR, but couldn't see in TODO

@justcoon
Copy link
Copy Markdown
Contributor Author

justcoon commented Jan 1, 2024

@afsalthaj

Firstly, its not scalable idea to reuse services. I hope we don't have common Service types. You mentioned TemplateService is duplicated, and I believe it should be.

they are "duplicated" because there are differences, i like to avoid "duplication", but do not have idea now how to do it,
more like we did custom rest APIs impl. also as there are differences

@justcoon
Copy link
Copy Markdown
Contributor Author

justcoon commented Jan 1, 2024

@afsalthaj

Also , I believe, if we have dedicated packages already for template , worker etc (and similarly in golem.cloud package, we have project, account etc) their corresponding types should also be in the same package, instead of scattering these types into golem.common (and golem.cloud.common). May not be for this PR, but couldn't see in TODO

sure, i agree that it making more sense to have project related types in project related package, we can definitely discuss it in team and make some decision

@vigoo
Copy link
Copy Markdown
Contributor

vigoo commented Jan 4, 2024

I used this tool previously (even wrote an sbt plugin to wrap it: https://github.com/coralogix/sbt-protodep): https://github.com/stormcat24/protodep which we probably could use to store the protos in separate repos and depend on each other. (I never used it in a way that one set of protos depended on another, just to share the protos between multiple service repos)

@vigoo
Copy link
Copy Markdown
Contributor

vigoo commented Jan 4, 2024

Maybe instead of having two TemplateServices, they could have a different name (not just namespace), to make it easier to work with them

@vigoo
Copy link
Copy Markdown
Contributor

vigoo commented Jan 4, 2024

I also would do the refactoring of worker-executor-base to get rid of account id and resource limit handling as a separate PR (as you proposed)

@vigoo
Copy link
Copy Markdown
Contributor

vigoo commented Jan 4, 2024

Regarding duplication of service interfaces, I think I like @justcoon 's approach more: to keep as much un-duplicated as possible and only split where necessary, instead of just duplicating everything and have two completely distinct set of proto files. So in general I like this PR

@afsalthaj
Copy link
Copy Markdown
Contributor

For an OSS dev, the name TemplateService makes easier sense. So if we are having explicit names I guess I personally prefer that to be done in cloud repo.

May be for closed source, I suggest CloudTemplateService (CloudWorkerService etc) so that IDE can list with all services that are part of Cloud. This idea can be applied to all services. I initially didn't change naming because I couldn't reach a conclusion on what to. Now it's time.

@vigoo @adamgfraser @justcoon

@vigoo
Copy link
Copy Markdown
Contributor

vigoo commented Jan 5, 2024

Regarding moving data types to different packages, I'm OK with that, although I think there are some which does not necessarily belong to only a single service - those could remain in some shared place like now

@justcoon
Copy link
Copy Markdown
Contributor Author

justcoon commented Jan 5, 2024

I added Cloud* prefix for grpc cloud services,
more like I agree that Cloud* prefix, will make it easier to search, but also at the other side as there are different namespaces, it is not needed,
personally I think we should not end in state that we will try to put "cloud" to every type for easier identification ..

@afsalthaj
Copy link
Copy Markdown
Contributor

GolemError was refactored to golem.worker.WorkerExecutionError

Don't know if this creates lots of merge conflicts. Module imports and renamings during merge conflict resolution can be a super painful task.

Not sure if we need renaming as part of this PR. But not strongly opinionated. cc @justcoon @vigoo @adamgfraser

@afsalthaj
Copy link
Copy Markdown
Contributor

From an OSS development/contribution perspective, developer friendliness is something super important.

If I want IDE to list all workerExecutuonError, then prefix is better than suffix

Also suffix Error looks a bit overly verbose
Just import worker.error package, and get InvalidWorkerExecution - or something along those lines. These flattened arbitrary error namings , has been personally very difficult for me to comprehend.

@justcoon
Copy link
Copy Markdown
Contributor Author

justcoon commented Jan 8, 2024

@afsalthaj

GolemError was refactored to golem.worker.WorkerExecutionError

Don't know if this creates lots of merge conflicts. Module imports and renamings during merge conflict resolution can be a super painful task.

there are already changes anyway, now it can take more time to revert it here, then apply it in golem cloud ...

# Conflicts:
#	golem-cloud-server-oss/src/grpcapi/mod.rs
#	golem-cloud-server-oss/src/grpcapi/template.rs
# Conflicts:
#	Cargo.lock
#	golem-worker-executor-oss/src/context.rs
#	golem-worker-executor-oss/tests/common/mod.rs
# Conflicts:
#	golem-cloud-server-oss/src/grpcapi/template.rs
#	golem-worker-executor-base/tests/common/mod.rs
@afsalthaj
Copy link
Copy Markdown
Contributor

Approved

@vigoo vigoo merged commit 96cebe6 into main Jan 19, 2024
@vigoo vigoo deleted the grpc_api branch January 19, 2024 08:27
vigoo added a commit that referenced this pull request Dec 10, 2024
vigoo added a commit that referenced this pull request Dec 10, 2024
GOL-225 Update cargo component deps automatically
vikashsprem pushed a commit to vikashsprem/Test that referenced this pull request Feb 18, 2025
vigoo added a commit that referenced this pull request Aug 15, 2025
vigoo added a commit that referenced this pull request Aug 15, 2025
GOL-225 Update cargo component deps automatically
vigoo pushed a commit that referenced this pull request Aug 17, 2025
vigoo pushed a commit that referenced this pull request Mar 12, 2026
Fresheyeball added a commit to Fresheyeball/golem that referenced this pull request May 12, 2026
CONTRIBUTING.md golemcloud#8 turned out to be unsatisfiable in the sandbox.
The cli integration test suite has exactly two modules:

- `bridge_gen::*` — generates wrapper code at test time, then
  `npm install` / `cargo build`s it. No pre-vendorable deps because
  the manifests don't exist until generation.
- `app::*` — scaffolds mixed TS+Rust apps and runs `golem-cli build`
  on them; same dynamic-deps shape.

Filtering both leaves zero tests, so the check would be vacuous
(\"0 passed; 0 failed; 113 filtered out\"). Removed in favor of
documenting that it's a local-only check. Everything else from
CONTRIBUTING.md is wired.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants