docs: Managing microservices dependencies first draft

[john-gom]

What

ADR for service dependencies

TBA: We should proably create a team for people who should review these and add that to the CONTRIBUTORS file

[alexgarel]

So far, good to me :-)

[john-gom]

Hi @raphael0202, @alexgarel , @stephanegigandet, @teolemon I've added a ports section which is something we discussed a while back. It would be good to go back to existing projects and adjust the port numbers to avoid clashes and add them to this list

[alexgarel]

@john-gom there are services where we don't necessarily need other projects running to develop. This is the case for example for robotof, where we just need to get data for the mongoDB, same for search-a-licious you just need to import some ES data. But you might need it all to test integration.
Should we also formalize the way we do it?

[teolemon]

@alexgarel (typed with Neuralink™)

[john-gom]

Hi @alexgarel , yes there are different levels of dependency. If we have hard dependencies like mongodb and redis then maybe these should be in their own separate repos? Certainly something to discuss some more...

[hangy]

One issue that I noticed when testing the PoC in the keycloak branch: If a project (in this case openfodfacts-auth) has a dependency that the calling project (here openfoodfacts-server) doesn't require (here: node.js to run node build-scripts/build_languages.mjs), then this implicitly becomes a direct dependency of the project where make run is called from. I have a fear that this won't work well in more complex scenarios.

I don't know if this works with all projects, but one option might be to require using docker run or docker compose run, like we do when npm needs to be used in ProductOpener when running targets like front_build. Otherwise, I fear that the plethora of dependencies that one needs to install locally will be way too difficult for people to easily jump into development.

[john-gom]

One issue that I noticed when testing the PoC in the keycloak branch: If a project (in this case openfodfacts-auth) has a dependency that the calling project (here openfoodfacts-server) doesn't require (here: node.js to run node build-scripts/build_languages.mjs), then this implicitly becomes a direct dependency of the project where make run is called from. I have a fear that this won't work well in more complex scenarios.

Hi @hangy , there shouldn't be a need to run any scripts locally when dependencies are pulled in, I'll have to investigate what's going on if that is the case with openfoodfacts-auth

[hangy]

there shouldn't be a need to run any scripts locally when dependencies are pulled in, I'll have to investigate what's going on if that is the case with openfoodfacts-auth

Don't worry. Maybe I mixed something up or made a mistake. Will have another look tomorrow!

[john-gom]

Hi all, I've had an idea regarding not loading dependencies into the parent folder. We could introduce a "DEPS_DIR" environment variable and if this is not set then dependencies will load into a "deps" directory of the current project, thus avoiding going to the parent of the current directory. If someone wants to develop on multiple projects simultaneously then they could set the DEPS_DIR to the parent of their working directories. Code might be something like this:

load_dependencies:
	@if [ -z "$$DEPS_DIR" ]; then export DEPS_DIR=$$PWD/deps; fi; \
	mkdir -p $$DEPS_DIR; \
	for dep in "project1" "project2" ; do \
		if [ ! -d $$DEPS_DIR/$$dep ]; then \
			git clone --filter=blob:none --sparse \
				https://github.com/openfoodfacts/$$dep.git $$DEPS_DIR/$$dep; \
		else \
			cd $$DEPS_DIR/$$dep && git pull; \
		fi; \
		cd $$DEPS_DIR/$$dep && make -e run; \
	done

If there is recursion then the recursed project will still be cloned into the deps folder but it won't continue recursing because the "{PROJECT1}_RUNNING" variable will be set.

It does make it a little bit more complicated, but at least people who are just working on one repo won't see side effects in the parent of their working folder and might avoid issues with online development environments like codepages.

What do other think? cc @hangy @alexgarel @stephanegigandet @raphael0202 @teolemon

[alexgarel]

We could introduce a "DEPS_DIR" environment variable

Good idea @john-gom

If someone wants more granularity, he can even use symlinks in the deps/ folder to reuse an existing directory.

[alexgarel]

Great, thanks !

Just two small remarks.

[alexgarel]

This will also affect ordinary commands, won't it ?

[alexgarel]

If I understand it well, in off-server we have to define a OFF_SERVER_PROJECT_NAME ?

[john-gom]

Not yet. It is only a problem when you are working on a project that needs to be pulled in as a dependency as you don't want any environment variables from the depending (parent) project to overwrite your own.

[alexgarel]

Cool !

[teolemon]

Do you have the source for the chart. Can you put it there, or a reference URL ?

[teolemon]

https://docs.google.com/presentation/d/1N6pgEFQ7s9kAeN8Y53GHBBjADdxA1BqTghEAFgY6CsY/edit#slide=id.g211e97e0566_0_0