Same Name Services on Different Databases - Foxx Bug

[p30arena]

my environment running ArangoDB

I'm using the latest ArangoDB of the respective release series:

• 3.1.21

Mode:

• Single-Server

Storage-Engine:

• rocksdb

On this operating system:

• Linux
  • CentOS 7 .rpm

Foxx

These are the steps to reproduce:

1. send a req to http://127.0.0.1:8529/_db/yabco/yabco-service/rpc
2. send a req to http://127.0.0.1:8529/_db/yabco-dev/yabco-service/rpc
3. send a req to http://127.0.0.1:8529/_db/yabco/yabco-service/rpc
4. the data is stored on yabco-dev instead of yabco
   ...

yabco-service on yabco, is running in production mode.
yabco-service on yabco-dev, is running in development mode.

0- create two services with mount points of /yabco-service on two different databases with different users, say yabco and yabco-dev.

---

To make sure its not from same name

To make sure that the error was not from me, I changed the second service name from yabco-service to yabco-service-dev.

now on yabco I have yabco-service
and on yabco-dev I have yabco-service-dev

UPDATE

Even changing the service name doesn't help!
I have to go and switch the service from prod to dev and again from dev to prod mode, so to make the data get stored in the correct Database!
What a nasty BUG! 👎

@jsteemann @mpv1989 @pluma @fceller @hkernbach

[p30arena]

Why nobody answering?

[jsteemann]

You mentioned the following sequence of operations to reproduce the problem:

send a req to http://127.0.0.1:8529/_db/yabco/yabco-service/rpc
send a req to http://127.0.0.1:8529/_db/yabco-dev/yabco-service/rpc
send a req to http://127.0.0.1:8529/_db/yabco/yabco-service/rpc
the data is stored on yabco-dev instead of yabco

However, what is each request to "/yabco-service/rpc" supposed to do? Store a document in a collection? How is that collection accessed exactly (e.g. by what means and what code)?

Does the last line "the data is stored on yabco-dev instead of yabco" mean that for the third request the data is stored in "yabco-dev" whereas it should rather be stored in "yabco"? Or do yo refer to any data being stored in "yabco-dev" (which is probably expected as one of the above requests was sent to "yabco-dev".
Can you please elaborate?

[p30arena]

I'll explain a situation that happens when we start the dev server (a single req sent to yabco-service-dev is enough to reproduce the bug),
we are working on two separate ports, say: A.com:80 and A.com:10000
A.com is our production site(DB: yabco, SERVICE: yabco-service), and A.com:10000 is the DEV site (DB: yabco-dev, SERVICE: yabco-service-dev).

When I login, a session with a token is inserted into the respected database.

I have a user, named ali, on both yabco and yabco-dev.

I run the A.com app server. a single req is made to yabco-service for some initializations.
Then, I run the A.com:10000 app server. a single req is made to yabco-service-dev for some initializations.

I try to login into A.com using ali,
Login is successful, the page is redirecting to the dashboard, but suddenly goes back to login page,
what's happening?

I check the token stored in the cookie, it is not stored in yabco, it is stored in yabco-dev!!!
I go to services pane, find the yabco-service, switch it to development mode, then switch it back to production mode.

now I go back and login again, now I'm successfully in the dashboard page.

[p30arena]

I've also added some logging to my services both on yabco and yabco-dev.
when I'm on the A.com, I receive all the logs coming from yabco.
so there's no wrong routing in the proxy server.

[p30arena]

I captured a video to better demonstrate the situation:
https://youtu.be/YxsD8xIEO_c

[pluma]

@p30arena That you're seeing the same problem even when you rename the service leads me to believe there's something else going on with your setup.

Can you please try the following:

• Disable the A.com:80 server/proxy and perform the requests against A.com:10000

• Disable the A.com:10000 server/proxy and perform the requests against A.com:80

There are numerous reasons I don't think the described behaviour could occur in ArangoDB itself (especially the part where documents end up in the wrong database) and I'm fairly confident the problematic behaviour originates either in the browser (e.g. AJAX against the wrong domain) or in a misconfigured proxy.

However if this really is an issue in ArangoDB it's difficult to diagnose it further without a minimal test case we can explore. Based on your description I would assume the problem to persist in the following scenario:

1. Install a service in (non-system) database A
2. Install an identical service in (non-system) database B
3. Switch service in B to development
4. Perform request against service in A
5. Request is erroneously handled by service in B
6. Switch service in B to production
7. Perform request against service in A again
8. Request is handled by service in A as expected

@mpv1989 Can you maybe give the above steps a try? I know development mode affects the routing, so I wouldn't completely rule out there's something wonky going on inside Foxx.

[p30arena]

@pluma I did what you asked for, the issue is not related to the proxy, as I said before I placed logs on both proxies to check if anything goes wrong!

Please take note that yabco-service is running in production mode but yabco-service-dev is running in development mode.

here's the video for the POC:
https://youtu.be/uji90IXQBG4

[p30arena]

@pluma is it related to the Manifest??
I checked, and I saw that in the service manifest, the name is yabco for both services (problem with copy pasting...).

This is the manifest for both the yabco-service and yabco-service-dev:

{
    "name": "yabco",
    "version": "0.0.0",
    "description": "YABCO SERVICE",
    "engines": {
        "arangodb": "^3.0.0"
    },
    "author": "Ali",
    "license": "Apache2",
    "main": "index.js",
    "scripts": {
        "setup": "scripts/setup.js",
        "teardown": "scripts/teardown.js"
    },
    "tests": "test/**/*.js"
}

when I switch from prod to dev and back again, the service reads the manifest.
the names may conflict.

[p30arena]

I changed the manifests and again did the steps that @pluma asked.

[root@ip42 yabco]# cat /var/lib/arangodb3-apps/_db/yabco/yabco-service/APP/manifest.json
{
    "main": "index.js",
    "scripts": {
        "setup": "scripts/setup.js",
        "teardown": "scripts/teardown.js"
    }
}
[root@ip42 yabco]# cat /var/lib/arangodb3-apps/_db/yabco-dev/yabco-service-dev/APP/manifest.json
{
    "main": "index.js",
    "scripts": {
        "setup": "scripts/setup.js",
        "teardown": "scripts/teardown.js"
    }
}

The problem still exists, I'm confused 😕
Note #1: the login and checkLogin APIs use POST method, so the browser wont cache them.
Note #2: I never use the db name in my service codes:

[yabco@ip42 APP]$ grep -rnw '.' -e 'yabco'
./README.md:1:# yabco-foxx

[pluma]

Okay, that's pretty disturbing, especially because neither the db nor the mount paths are the same. The "name" field in the manifest should be irrelevant as it's mostly cosmetical.

Thanks for confirming this with a minimal example. Can you clarify how you access the collection? Do you use the arangodb.db object or the module.context.collection method?

[p30arena]

@pluma

const db = require('@arangodb').db;
const usersCollection = db._collection('users');

and I use the router:

const router = new require('@arangodb/foxx/router')();
router.get('/doc', function
router.post('/', function
module.context.use('/rpc', router);

[jsteemann]

@p30arena : is the assignment

const usersCollection = db._collection('users');

done outside out the routes?
If yes, then the following may happen:

• the script is loaded when first accesses. This will bind the collection "users" from the currently accessed database to variable usersCollection.
• when the scripts is accessed for a different database but has been cached already, the const assignment will not happen again. Instead only the route function is executed, but it will refer to the already assigned variable, which will then point to the wrong collection.
  The solution for this is potentially to make the variable assignment for collections inside each route, and not in the global script scope.
  Does this help?

[p30arena]

@jsteemann that's exactly what we do

function createModuleHashTable(path) {
	return fs.listTree(path)
	.filter(e => e.endsWith('.js'))
	.reduce((a, b) => {
		a.set(
			b
			.replace(/\//g, '_')
			.slice(0, -3)
			.toLowerCase(),
			require(fs.join(path, b))
		);
		return a;
	}, new Map());
}

then in the route we call the function.

global.import_model = (n) => { return ModelsHashTable.has(n) ? ModelsHashTable.get(n) : null; };
...
router.post('/', function (req, res) {
...
const model = import_model(model_name);
...
}

If thats the case, it must be a bug related to Foxx, they are two services on two different databases, with two separate memory allocations, why should my objects interfere?
.
what kind of caching are you using?

[pluma]

@jsteemann That seems unlikely. Foxx service files (including bundled dependencies) are cached per service per db, not globally like other JS files. It's not necessary to create the collection reference per route, in fact none of the Foxx examples do this.

@p30arena wait, you're redefining something on the global object? That might be the problem. As both of your services define the same global variable, only one of them can win. This will always be the one mounted last, so it'll usually be the development service because that is loaded on every access whereas the production service is only loaded once per context on first access.

In other words: all instances of the service (in the same context but across different databases) will share the reference global.import_model of one instance of the service. So you're effectively exposing access to one service's collections to all instances of the same service on any database. This is why your documents end up in weird places.

This also explains why the mount path is irrelevant. I suggest you don't pollute the global object. The kind of meta-programming you're doing with dynamic requires in a lookup table is already a bit dubious (e.g. your code will likely fail on Windows systems) but if you really want something "global" to all your service files but not the database, consider using module.context.service instead of global. This is the service instance.

[pluma]

We should probably add a warning about modifying "global" in the documentation somewhere. Probably in the incompatibility caveat section (which currently lives in the chapter on dependencies but should probably go somewhere else).

[p30arena]

@pluma that's weird, why should the global var be accessible to all of the services?!
Ok thats not a bug from your point of view, but thats a Defect from my point of view.

[p30arena]

@pluma I fucked everything by using this line:
global = module.context.service;
Its really hard to change the codes, no workaround?
😄

[pluma]

The "global" object is global. There's no way around this.

FWIW the recommended approach would be to extract your model map into a separate module you can require wherever you need it and then use that. Polluting the global namespace is generally discouraged in any JavaScript environment as far as I'm aware.

EDIT: Try search and replace maybe?

[p30arena]

(I have no time for refactoring right now, this is not a good solution, but it works)

I placed all global vars in exp;

module.context.service.m_tools = exp;

module.context.service.lts = `var {
	m_ok,
	...
	UNIQUE_CONSTRAINT_ERROR,
	NO_CREDIT,
} = module.context.service.m_tools;`;

now in another module, I use this code to define vars:

eval(module.context.service.lts);
console.log(NO_CREDIT);

[pluma]

I really think you should be putting these in modules and require them as needed. I also think you shouldn't use eval: https://stackoverflow.com/questions/86513/why-is-using-the-javascript-eval-function-a-bad-idea

Using module.context.service or global is a hack. It's not a good way to design your application.

[dothebart]

ArangoDB 3.2.2 containing this bugfix is available for download - thanks for reporting.

Closing.