Registries 5.0 Proposal #44
Comments
tsvetlin
added
5.0
Core System: Service Registry
|
Great initiative to post a proposal before actually digging in! After skimming through everything and reading through sections that popped out to me, I make the following suggestions:
If you want me to look at any other things in particular, please tell me where to look. Thanks again! |
|
Application Registration Process in high definition
|
|
|
Trail ID is now part of the proposal #15 |
I approach my review of your proposal solely from the perspective of what I think you believe about what messages are going to be sent to and from your implementations. It may be a suitable choice for your implementation to only support X.509 and MAC addresses, but the Device Registry protocol must be able to support other cryptographic standards (e.g. PGP certificates, the direct use of raw public keys, or Kerberos) and network interface identification mechanisms (e.g. the ID byte of CAN bus). I'm sorry for probably being a bit overly pedantic, but I just want to make sure there is no confusion about that Arrowhead itself is able to, at least theoretically, support any computer communication mechanisms. By sticking to X.509 and MAC in your implementation, we must not make it seem as if those two are the only that can be used with Arrowhead. Other implementations could be designed to support other combinations of standards.
I'm still eagerly waiting for @vanDeventer to present his proposal for the service registry operations and messages (or did the assignment change without me knowing?). What he would like the field to be called matters (to me at least). The field name either being the same in the database as it will be in the actual messages or sticking to a pattern should make it appear the least confusing (at least in my mind). Do as you please, of course.
This is something that is not explained in any detail in the Concepts Reference, but that I'm planning for the (now delayed) Documentation Reference. There is, of course, an important distinction being made between service instances and service types (or definitions, as you name them). An instance is something tangible you can actually contact, while a type describes how you communicate with an instance adhering to it. The type is only relevant to consider if you do not already know how to interact with services that implement the type. Most systems will be hard-coded to interact properly with certain service types. To those systems, it will not matter if the service registry knows the type details (such as what operations they have). It only matters that these systems can identify the type identifiers of the instances they communicate with. Storing service type details in the service registry could perhaps be a boon to code generation, etc. That being said, I do not believe we should require that every single system registering a service should be forced to ensure the service registry knows of the exact type details of each of its services. To be frank, I don't believe the service registry should store service type details at all. In my mind, the service registry maintains runtime information (what is running where?). Service type details is "planning time" information. That should go into something more akin to the "Plant Description" system. Runtime information tends to be "hot" (performance critical) while "planning time" information tends to be "cold" (not performance critical). |
|
@emanuelpalm Nice to see comments from you. I agree with you that @tsvetlin 's proposal is the correct approach to get what we decided in the Roadmap workgroup. “Beauty is no quality in things themselves: It exists merely in the mind which contemplates them; and each mind perceives a different beauty.” ― David Hume, Of the Standard of Taste and Other Essays I am very much interested in what are in these messages. I do not know enough to make a good proposal and could go with the one you made earlier. #22 (comment) I agree very much with you that the name of the fields is very important. end_of_validity is a confusing name and expires_at is much better. In my current Go implementation, the Service Registry System is the one which fills that timestamp and the service provider is to renew/update its registration. (interestingly enough, there is no PUT method with /register). The duration between updates is now fixed, but I would have preferred that it is suggested by the provider system and requested by the consumer system. [ia consumer knows how stale the service registry can be for its own purpose; an if a service expires, it could trigger a future push orchestration] So, should duration be a field. My interest here is the system of systems' recovery of device and system failures. When/Why does an application system need to know when a system or service was registered? Do these timestamps need to be in the message? My work has not touched cybersecurity yet but the hardware I have in my office has hardware based private keys (e.g., https://www.st.com/en/secure-mcus/stsafe-a110.html) which is what I plan to use to authenticate the devices. I do not have much to say yet about the MAC address except that the router in my office pretends to be my computer so that I can have other devices... In any case, I agree with you that we need to define the messages for all three new systems. |
While Arrowhead theoretically can support every cryptographic standard, in reality they have to implemented. We have to start somewhere the implementation, therefore we propose X.509 to be the first in line to be implemented, whilst it can be followed by other standards later. In the
Ok.
If everything is in the documents, then, why use Service Discovery, why Orchestrate??? You can just hardcode everything. Authorization most likely will be using NGAC, and it will support fine grained access, to the operation level. For this reason, the operations must be available in the service registry too. |
|
Two comments on the PDF.
If there is telco on the topic this week, I somehow missed the invitation. |
|
Thank you for clarifying your position. It was plenty of news to me.
Alright, since we haven't decided what these messages should look like, and no one seems to be ultimately responsible, then I propose that @tsvetlin (or someone else at AITIA) adds those messages to his proposal. I'm personally much more interested in the messages than the implementation, even though the implementation is certainly not uninteresting. Please look at my proposal for a service registration message #22 (comment) Jan mentioned, which I think is something that is approaching a good message layout. I haven't really thought of how to design the device and system registration messages yet, so feel free to try out whatever layouts you think could work.
Yes, exactly!
I'm sorry for being confusing, but as I said already (but perhaps being a bit unclear), my concern is about messages, not about database tables. You may decide to have the table fields however you want. Be aware, however, that you may be asked to support other standards in the future, such as OpenPGP (even though it may be a bit unlikely, as OpenPGP certificates were deprecated in TLS version 1.3 (look towards the end of Section 4.4.2)). MAC addresses, X.509 and TLS are likely to remain the dominant standards for years to come, but there are on-going attempts to replace them (e.g. Information-Centric Networking). It may be more sensible to hard-code support for TLS/X.509/MAC now, and change the table if other contenders ever become relevant. I like the way how the table changes communicate to others working on the core-java-spring systems that support for other standards may be added.
Only service types are described in these documents. Not the service instances. We must still depend on Service Discovery and Orchestration to find instances of the service types we want our systems to communicate with. Finding a service of the correct type only requires the service to be associated with exactly one service type identifier. If we want to dynamically analyze services at runtime, then information about the operations becomes a requirement, of course. Automatic service validation, which you mention, may be a valid use-case for that. I don't think it should be a mandatory part of the Service Registry system, however. Not everyone will want such validation, and I believe there is going to have to be plenty of room to configure such a validation system. I still believe that information about operations should not go into the service registry, but into another system. I understand that I don't always get what I want, however. ;-)
It is true that NGAC will provide fine-grained orchestration rules. It is also true that automatic validation of NGAC rules may require detailed information about service operations. It is not true, however, that the service registry must store that information for the validation to be possible (another system could store it). Neither is it a matter of-course that everyone will want to perform such validation. The NGAC rules themselves will be executed on every system providing services, not by the service registry or authorization system (other than perhaps to regulate access to their own services). They do not have to be validated before being used. Also, it may not be relevant to validate all rules or all operations, even though some of them are. This is why I don't like you enforcing that every system registers the data types and operations of its own services. |
|
"To allow that the databases can be handled by same database server"
Metadata
The echo service
Register-device
Register-service
Two staged startup
name (mandatory): name of the device. Text. Maximum length is 63 characters, only contains letters (english alphabet), numbers and dash (-), and must start with a letter (also cannot end with dash). Must be unique.
mac_address (mandatory): MAC address assigned to the device in text format:12 digits hexadecimal number with colon every two digits (an octet). Must be unique.
DeviceRegistry
GENERAL COMMENT ON THE TABLE
|
|
In order to fully achieve loose coupling between core services (making them replaceable by e.g. not sharing db's), we see a need for a new field, to be added to both the request and the response: serviceInstanceId to be added to: To be used e.g. in /authorization/intracloud/check identifying the service instance to be checked. |
|
Hi, Every system (provider or consumer) needs an interface or interface to communicate with other systems. Therefore, the interface_protocol parameters should be part of the System registry, not the Service Registry. If we keep the interface details attached to services we are limiting the interface information to providers and neglecting consumers. In addition to this comment, I consider that the information added in the metadata field is probably not enough. If we consider other interface description languages ( for example OpenAPI or WADL) the data is not structured in pairs but in more complex files. Which I think, maybe I am wrong, are not supported in this version either. Thank you for your work in creating the document and proposal. |
|
@UlfSlunga-Sinetiq As far as I know, we will add unique service instance identifiers to the messages of relevance in the v5.0.0 release. You are not the only one who have raised this concern. @CristinaPaniagua @tsvetlin I would like to have some critera for what information should or should not go into a given system. Otherwise we risk them holding way too much of it? I will try to think about a suitable criteria. How about them only holding runtime information? What exists in the cloud and not what those things are. |
|
Hello, I would like to add some things to the discussion both on a general level and in detail.
// David (Sinetiq, I used to be at BnearIT ~3 years ago for reference) |
|
MoM from Roadmap v5.0.0 meeting March 4.
Backward compatibility
Independence between Service Systems and Device registries
Echo or Monitoring service Next meeting will be Wednesday March 16 at 15.00 ordinary Roadmap WG time-slot. Primary agenda point will be to finalise the issue #44 comments not yet addressed starting with Register-device from @jerkerdelsing The ambition is to close all points next meeting such that development of detailed documentation and subsequent coding can start. |
|
Regarding multiple operations of services: To me, the point of SOA is that each service should provide one thing and that that one thing should be limited in scope. From the meeting today I feel like I've misunderstood something about services, could someone clarify? |
|
@ajoino The idea was always part of the Arrowhead Framework, but was implemented in a "bad" way.
functionality. These functions were named operations recently. In 3.X, 4.X, these functions were implemented as a separate service. I believe this aproach is also viable and good. On a roadmap meeting we agreed, that it has to implemented the "correct" way, so services will have at least one operation from now. Yes it may be confusing, but in the end nothing really changes, only there is one more abstraction layer. |
|
Thank you @tsvetlin! It's just a matter of personal preference, and I don't feel very strongly about it. |
|
Generally SOA is very permissive on these kind of questions. A couple of things have though been defined with Arrowhead since quite some time
We currently have no upper bound of operations stated but in a micro service context the number is presumed to be rather small. |
|
@jerkerdelsing perhaps I'm misunderstanding exactly what we mean by 'operation'. My worry is that by having services whose operations are based on REST methods, we will make it harder to provide services using protocols that are not RESTful. I'm sure we have thought of this and I'm just out-of-the-loop currently, just wanted to raise my concerns. |
|
@emanuelpalm @tsvetlin I am trying to get a better understanding of what the payload to the device registry should look like. I see now that I omitted the authentication string. My first attempt results in I get this when I run host.go program on my Mac while being connected to the ethernet and WiFi. |
|
Since I am not agnostic, I am having issues right now. I did not work with MQTT nor web sockets and my problem is to understand the difference between operations and services... I will start with an example: I have two motorized valves used in a heating system. I can request their current positions and request that they update their position (from 0 to 100%, with 100% meaning fully open). One valve is in the kitchen and one bathroom. What should my path look like with the different protocol? |
Service vs. Operation@ajoino @vanDeventer If we make a parallel with how Object-Oriented Programming works in Java, you may think of a system as a class instance (i.e. an object), a service as a Java interface and an operation as a method. A major difference is that in Arrowhead, our "class instances" (systems) cannot have methods without declaring them as part of a "Java interface" (service). Systems maintain internal state that can be queried and/or updated via the operations of the services that system hosts (or implements in Java lingo). A service provides an interface (point of communication) through which a particular task can be fulfilled. If that task requires multiple messages to be properly fulfilled (e.g. maintaining an accurate entry in a service registry), then that interface will have more than one operation. By task i mean any kind of value-creating activity. |
|
@vanDeventer An operation is something that accepts exactly one message and replies with exactly zero or one messages. In REST, an operation would be identified by an HTTP endpoint, formulated as an HTTP Method (POST, PUT, DELETE, etc.) and a Path (e.g. In MQTT, that could be a specific topic, which looks as the path in REST. A major difference, however, is that MQTT does not have or require the use of any methods (PUT, POST, etc.). Neither does it demand that the path emulates a directory structure (as does the REST spec.). The above example HTTP endpoint identifier could be formulated for MQTT as WebSockets is just a streaming protocol, just as TCP, possible multiplexed with HTTP messages. You would have to define your own protocol on top of it for us to know how to formulate an Arrowhead operation in that protocol. You could, for example, send JSON-RPC or CoAP messages through WebSockets. In the former case, an operation could be a namespaced function name like |
I agree that a document with API details only would be better to base this discussion on. @tsvetlin said during our last telco that such a document will come later.
I agree. That would have been nice.
We had a discussion on this on the telco. There is a trade-off being done here. Either we optimize for fewer interdependencies and fewer messages (as you suggest), or we optimize for less data duplication (i.e. not having the same IP addresses stored in both the system and the service registries). Less data duplication means it becomes easier to guarantee data integrity, which is important. My impression was that most (including myself) were in favor of the former of the two trade-offs.
I had a discussion with @vanDeventer about this a while ago. The highest level of flexibility becomes possible if all information to contact a particular service is in its service registry entry, as a propose here: #22 (comment).
@tsvetlin Would have to answer this question. |
|
@emanuelpalm thank you for the description of service vs. operation. The point of disagreement is that I consider the services themselves analogue to methods and don't see the need to involve another level of abstraction.
Maybe there is a way to reconcile these ways of thinking (I think we were getting there during the telco): |
Let's say you are writing a system that will negotiate contracts with other systems ;-) . In order for your system to be able to negotiate properly with some other system, three kinds of messages will have to be passed between them: (1) proposals (and counter-proposals), (2) acceptances and (3) rejections. Both your system and the system of the counter-party host the same service, able to deal with these three messages. Now, as a service designer, you currently have two options on how you want to design this:
Both of these approaches would work, of course, because what fundamentally happens at each interface level (network, system, service and operation) is that the message is looked at and passed on to more specialized code until the code that can actually handle it receives it. In the two examples above, that "looking at" occurs inside the service in both cases. However, in case 2 the implementor (programmer) must make that discimination manually (i.e. a "switch" statement calling internal functions). My perception is that most kinds of exchanges will involve different kinds of messages, which means that you generally land on a cleaner implementation if a distinction is made between services and operations.
Do you mean that the system registry should not store IP addresses? What do we want to store in the system registry anyway? |
|
@emanuelpalm
I think our disagreements are about what can be considered a service or not. For example, I would expect all services, whether HTTP or MQTT, to use the same identifier/endpoint/basePath/topic, whereas I take it you and many others think it's fine for a service to have many endpoints. AFAICT this distinction is a matter of taste and conventions.
No, I mean that the IP in your suggested service registration message is used for a different purpose than that IP registered as part of a system in the system registry, even if the IP is the same. Let me try to clarify:
The difference here would be how the information is handled.
I made this up quickly so there are probably a bunch of ways to make this more efficent and possibly remove the system registry from the process. But my point is that service registration data is for establishing connections, which is, in many ways, completely different from system data which describes the local cloud, independent on the services and what connections they can establish. So while the data looks the same, it's semantically different and not duplicated. |
As things are right now, systems may or may not provide their services, while a service must expose all of its operations or not be provided at all. If we would implement the message receivers as services, a given system would be free to only provide one or two of those services. However, the three services are only useful if all of them can be used at once. Guaranteeing that they are all available is important. If we would remove the operation layer and do as you propose, I guess we are essentially removing the system layer and renaming services to systems and operations to services. I like when things are as simple as they can be, so I reject the idea that this is a matter of taste. If we don't need one of these layers, I want it removed. I still have a hunch, however, that there is some fundamental reason why we cannot remove any of them. I just need to be convinced there is no such reason. Another way of viewing the system is as an abstract representation of a software instance. The software exposes services, each of which guarantees that a certain number of operations will be available. Isn't that a good enough reason to say we have to have three layers? |
|
Good morning, There is a French expression that describe my constant state with these ideas: «je comprends vite mais il faut m'expliquer longtemps » I have a few questions
|
|
Hi
See my comments below inline.
On Fri, 11 Mar 2022, 09:52 Jan van Deventer ***@***.***> wrote:
Good morning,
There is a French expression that describe my constant state with these
ideas: «je comprends vite mais il faut m'expliquer longtemps »
I have a few questions
1. Does an MQTT broker belong to a local cloud?
An MQTT broker do belong to a local cloud, even though the broker itself
is invisible in an Arrowhead context. The broker do not offer or consume
any services directly.
1. When you publish a topic to the broker, is is also registered as a
service with the Service Registry system? If yes, by whom?
The system that is publishing to a topic do not register anything.
Instead, the system that "owns" the topic and subscribes to it must
register it in the Service registry like it is done with http or CoAP.
There are some security concerns that I have outlined in the report on
enhanced MQTT security in an Arrowhead local cloud that I write earlier.
1. Do you use the GateKeeper and Gateway to subscribe to an MQTT topic
in another local cloud?
I didn't investigate this, but my intuition says that if inter-cloud MQTT
communication is desired, then the appropriate security mechanisms must be
used. And adding support for MQTT topics to the gateway/gatekeeper systems
makes a lot of sense to me.
Jens
…
—
Reply to this email directly, view it on GitHub
<#44 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA52PBKAK2FW45PSADUNW4LU7MCUNANCNFSM5PB52SHQ>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
|
Even though I think this is a bit outside the scope of this issue, following on what @jenseliasson said above.
If a service uses Pub/Sub over MQTT wouldn't the service producer register the topic then so consumers can consume through this topic? Maybe you referred to a non-pub/sub example though. Registering a topic as a producer is saying "I will announce changes in this forum, listen if you would like to receive them".
Using gatekeeper and gateway sounds reasonable. The gateway would simply consume one topic from one local cloud broker and then publish to another topic at a broker in the second local cloud. |
Sounds really good! |
This is not related to the authentication, but related to the Orchestration. During orchestration, with the orchestration flags a consumer can specify, that it only wants reachable results. The Service Registry checks if a requested provider is reachable, and only returns them at that scenario.
This is neccessary to circumvent the cross dependency of the Service Registry, System Registry, Device Registry. The Two staged startup enables these core systems to register themselves into each other and then finish the startup procedure to be available for other systems in the Local Cloud.
Yes! Unique in the LC. |
As @emanuelpalm said before, this will be included in 5.0.0 |
During the orchestration a consumer can specify which interface does it need, and the Orchestrator can find it a suitable Provider. What benefits do we have if we are aware what interfaces does a consumer support? Could you please further explain it with an example? |
URI Crawler is implemented in the JAVA core systems, to discover services necessary for their operation. The core systems are using the Service Discovery service's Query operation |
|
@emanuelpalm raised a concern, that the registration procedure (see quoted image) is complicated, many messages has to be passed. Circumeventing this issue. It can be a good solution to have a "Registry" supporting Core system, which could behave as an API Gateway, where the Application Systems could register/unregister themselves in one message. This would make scaling much easier (multiple instances of SR,SysR, DR) and behave as a load balancer.
Application System Registration Procedure: Gatherin data about core systems once, to handle incoming Application System requests.
|
|
@emanuelpalm in the last WG telco. You mentioned that you feel like SysR and DR are "almost mandatory" core systems. Could you please explain what do you exactly mean as "almost mandatory"? |
This could be a good solution. Could it be used to eliminate the call to the orchestrator as well? You send one message with your device, system and service details, as well as what services you are capable of consuming, and you get the appropriate service records in the response? I guess it should be possible to enable orchestration push in the message, as an alternative to receiving the service records in the first response. |
I assume that most use cases where an Arrowhead Local Cloud is employed will see great value in maintaining registries of what devices they have, what systems they are running and what services they provide. If not for practical reasons, but because that information is just useful in the general sense. This makes these registries "practically mandatory", or almost mandatory. However, each of these systems introduces another dimension of dynamicity. As far as I understand it, if you have a device registry the set of devices in your cloud can change at runtime. If you have a system registry the set of systems can change, while a service registry enables you to dynamically handle what services are provided and consumed. In other words, the more information about you local cloud that does not have to change, the fewer of these systems do you need (if you are willing to manually provide all systems with the information they need that would otherwise go into these registries). All this being said, the point of Arrowhead is dynamicity. If you remove dynamic service discovery the most vital type of dynamicity is lost. I guess you could modify the authorization and orchestrator systems such that they operate on predefined static service registries (via config-files or similar), which would make it possible to have late service binding without service discovery. Anyhow, I guess "practically mandatory" is kind of the same as "mandatory". |
|
Conclusion Two staged startup
Two staged startup seams approapriate Full common name (mandatory): name of the device. Text. Maximum length is 63 characters, only contains letters (english alphabet), numbers and dash (-), and must start with a letter (also cannot end with dash). Must be unique.
mac_address (mandatory): MAC address assigned to the device in text format:12 digits hexadecimal number with colon every two digits (an octet). Must be unique.
Name discussion in relation top X.509 certificate to be addressed in specific meeting. DeviceRegistry device table
Sinetiq will take the lead in creating SoSD, SysD and SD documents for a set of the core systems. First version by end of May 2022. Interface registration: provide a set of metadata which are recommended. To be further discussed in connection with the SoSD, SysD, SD documents as lead by Sinetiq, David Rutqvist comments
Szvetlin This and above MoM will be one of the sources for SoSD, SysD and SD document for the Arrowhead Architecture SoSD and ServiceRegistry, SystemRegistry, DeviceRegistry, authorisation and Orchestration systems, SysD and SD's. By this I close this issue. |
Dear everyone,
I would like to propose three core systems in Arrowhead 5.0, which are redesigned and built upon comments, feedback from the community.
Please read the attached PDF and feel free to express your thoughs, concerns, ideas.
A roadmap meeting will be held on 7th March, from 15:00-16:30 (CEST) do finalize this proposal.
Registries_5_0.pdf
The text was updated successfully, but these errors were encountered: