Stable IDs

[silverdaz]

According to Oscar, the stable IDs generation for CentralEGA is handled by a process at EBI and it depends on the content of the original file.

We use the generated stable ID after a successful ingestion, which allows us:

• not to waste any useful stable ID in case of errors (and no need to recycle or clean them up)
• mark the ingested file as completed, but not yet ready for data-out. Once the stable ID comes in, we flag the file as ready for download.

[silverdaz]

@blankdots : When you are back from holidays and read that, could you please give me a hand for the tests? (My local test worked fine).

The tests fail on the eureka, but I have neither touched the eureka code, nor its test.
So either it failed before, or there is some magic happening in the mock things.

[dtitov]

@silverdaz Could you, please, elaborate why not request the stable ID in the verify.py, right after successful checksum validation? What is the necessity of introducing new microservice?

In my mind, the logic of verify.py could be:

1. Validate checksum:
   a. If validation failed - do the same what we do in such case currently;
   b. If validation succeeded - do the same what we do in such case currently plus query CentralEGA for stable ID and update it in our database.

Will it work like that or am I missing something?

[silverdaz]

Ah, yes, sure, I had not thought about putting that code inside verify.py.

But this is asynchronous: We do not "query" CentralEGA and wait for the response synchronously.
We instead send them a message and they respond when they find the time.
That's why there is another service that listens to their answer, and that verify does not handle the stable ID.

[dtitov]

Are we making our system more complicated just to make CEGA's life easier? :)

To be honest, I would even include stable ID querying to the ingest.py, because I don't care much about wasting of some IDs: if we take a look at what UUID is, we can see that it's 128-bit long identifier, so you can imagine what a nearly endless space of IDs does it give to us.

And also I don't think that generation of ID is so much heavy and CPU consuming operation or that it's hard to implement: it's basically a call to UUID.randomUUID().

So, concluding, I would definitely not overload LocalEGA's setup with yet another microservice just because CentralEGA cannot or doesn't want to generate UUIDs for us upon request: neither them, nor we should be concerned about running out of ID's or about the heaviness of a synchronous approach.

Can you discuss this with Oscar and Sabela?

[jrambla]

@dtitov we are talking about Stable IDs, not UUIDs. StableID have a different life cycle that include tracking them for creation, updates (with versions) and eventual deprecation or obsolescence. Thus, we prefer to do not "waste" them, if possible.

[sdelatorrep]

I'm not sure about what we are discussing here: async vs sync communication and CEGA IDs vs LEGA IDs (=UUIDs)?
In my opinion, I don't see any benefit of using synchronous communication here. If CEGA is down what would you do? Retry until it is up again? What if there is a communication error? Retry again? How many times? And what happens if CEGA never answers? Retry? Or the other way around, what if LEGA requests an ID but something happens when answering back, what should CEGA do? Retry? We already have RabbitMQ in place, so let's use it :) It will guarantee messages are not lost and neither CEGA nor LEGA need this information immediately.
About using UUIDs instead of CEGA IDs, I guess it's another solution but with probably lots of consequences which must be considered.

[dtitov]

@jrambla Thank you for the clarification, I was not aware of StableIDs lifecycle - that, of course, makes things a bit more difficult.

At the moment, after your comment, I see two possible ways:

1. CentralEGA creates and provides us with the "StableIDs API" (probably the REST one) so that we can perform all kinds of the operations that you've specified: issue new IDs, version them, deprecate them and so on.
2. Or LocalEGA creates yet another microservice to handle that (basically, this PR).

In other words, one of the sides should become more complicated. And, obviously, as a representative of LocalEGA team, I would prefer not adding any extra complexity to LocalEGA's architecture if it's possible to avoid that.

@sdelatorrep I see your point and it makes perfect sense to me, but any communication channel has its own benefits and downsides. What if we send you a message, you receive it and after that, during ID generation, CentralEGA goes down for some reason (e.g. power outage)? Then we will never receive the message back because, after the restart, your side will simply "forget" about the fact that it actually needs to respond (no messages are queued - no state is preserved). In this rare, but still possible case, the archived file will never get its ID. And with HTTP request we at least can be aware of the fact that ID retrieval failed so that we can fall back to some backup mechanism.

Look, I propose another variant not only because of my personal preferences but more because of practical reasons that relate to our local infrastructural limitations. But, I guess, we should better discuss it tomorrow in a meeting. Anyway, thanks for the explanations.

[silverdaz]

What if we send you a message, you receive it and after that, during ID generation, CentralEGA goes down for some reason (e.g. power outage)? Then we will never receive the message back because, after the restart, your side will simply "forget" about the fact that it actually needs to respond (no messages are queued - no state is preserved).

This is what RabbitMQ handles. The message is not forgotten, and it will be (re)processed.

This PR is a solution to issue #260 .
There is always the possibility to make another PR/issue if you, @dtitov, feel that this solution is not suitable.
In the Agile spirit.

[dtitov]

@silverdaz I have read more about acknowledgments in RabbitMQ and yes, the case above can be handled.

[blankdots]

this should only be for local development (even more so that this is specific to @silverdaz env :) ), not particularly fond of having this sort of paths in the public repo (the same for other services)

[silverdaz]

ah... oups...I thought I had removed all of them...

[blankdots]

I am using https://www.python.org/dev/peps/pep-0257/ - this line should be imperative (the same goes for line 32 below).
see D401 http://www.pydocstyle.org/en/2.1.1/error_codes.html

I know we are not following pep8 and pep257, and it will take quite the effort to go over the previous code to format it following the rules, however I would like for the "newer" (a bit relative here what "new" means) to follow these guides.

[blankdots]

@silverdaz regarding this #341 (comment), you kinda touched eureka in the config (try and try_interval)
you can set them as env varisables in test using https://docs.python.org/3.6/library/test.html#test.support.EnvironmentVarGuard

[blankdots]

url is not valid, why not have http://localega.readthedocs.io/ there or the github repo url ?

[blankdots]

isn't qc queue deleted from defs.json ? Maybe a miss, or does it serve some purpose ?

[viklund]

I looked at the logic of the python script and I think it is ok.

I do think that the name mapper is a bit confusing. Maybe just, stable-id-reciever?

[blankdots]

ID Mapper is indeed not the best name.

Code has been reviewed. Changes to the code base imply changes to the deployment specified in these issues:

• Add new component ID Mapper (for stableIDs) LocalEGA-deploy-swarm#1
• Add new component ID Mapper (for stableIDs) neicnordic/sda-deploy-init#4