Draft: Create Network driver instead of directly using sockets. #330
Conversation
There was a problem hiding this comment.
I realize this is just an early draft, but this looks like a good start, working in the right direction. A couple of suggestions that may save time later:
- You may want to base your changes on the federated-cleanup branch of reactor-c, which will be merged soon, I think. It includes quite a few changes to the networking code.
- Our coding style uses spaces only for indentation, no tabs.
- Our coding style puts opening brackets on the same line and always encloses if statements in brackets when a new line is involved. E.g., instead of:
int netdrv_open(netdrv_t *drv)
{
if (!drv)
return -1;
return drv->open(drv);
}
it should be
int netdrv_open(netdrv_t *drv) {
if (!drv) {
return -1;
}
return drv->open(drv);
}
or
int netdrv_open(netdrv_t *drv) {
if (!drv) return -1;
return drv->open(drv);
}
- Header files should always document functions using JavaDoc style documentation.
|
@edwardalee Thank you for having a look. I'll keep in my what you mentioned. Especially for the indents and coding style, it was just the auto formatted. I'll fix it. Also I'll add up the documentation for header files. Thank you. |
…during every read.
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
Netdriver
Current LF code of the reactor-c uses TCP and UDP sockets directly, limiting to scale network options. This PR introduces a network driver to support various network types.
The
void* privwill be casted depending on the network type.APIs.
The main design is as a client and server. The RTI is always a server. Federates can be servers and clients. As a client, it should be able to connect to the RTI, and also connect to other federates. As a server, it should be able to accept connections from other federates for two reasons: decentralized coordination and physical connections.
So, the main design follows TCP socket server-client design.
The RTI will have a netdriver that accepts other federate connections, and a netdriver will be allocated for each federate.
The federate will have a netdriver to connect to the RTI as a client.
Next, a netdriver to accept other federate's connection as a server.
Finally netdrivers to communicate with each inbound and outbound p2p connections.
The description below separates the APIs as functions for clients, servers, and common each.
Common
netdrv_t* initialize_netdrv();
Initializes netdriver, and allocates memory.
void close_netdrv(netdrv_t* drv);
Frees memory allocated by netdriver.
int write_to_netdrv(netdrv_t* drv, size_t num_bytes, unsigned char* buffer);
This sends the buffer to the netdriver.
TCP: write()
MQTT: publish()
SST: encapsulate message with SST, and send.
The below two each has additional error handling.
int write_to_netdrv_close_on_error(netdrv_t* drv, size_t num_bytes, unsigned char* buffer);void write_to_netdrv_fail_on_error(netdrv_t* drv, size_t num_bytes, unsigned char* buffer, lf_mutex_t* mutex,char* format, ...);ssize_t read_from_netdrv(netdrv_t* drv, unsigned char* buffer, size_t buffer_length);
ssize_t read_from_netdrv_close_on_error(netdrv_t* drv, unsigned char* buffer, size_t buffer_length);void read_from_netdrv_fail_on_error(netdrv_t* drv, unsigned char* buffer, size_t buffer_length, lf_mutex_t* mutex, char* format, ...);Server
int create_server(netdrv_t* drv, int server_type, uint16_t port);
TCP: creates server socket, binds, and listens to server socket.
MQTT: Connects to broker. Subscribe to Topic “{Federation_ID}_RTI”
SST: creates server socket, init_SST(): loads keys
netdrv_t* establish_communication_session(netdrv_t* netdrv);
TCP: accept() for federate socket requests, return fed_socket.
Mqtt:
SST: accept() sockets. RTI does handshake with federates. SST_session_ctx is created for each federate.
Client
int create_client(netdrv_t* drv, int federate_id);
TCP: Creates a real-time TCP socket.
MQTT: Federate connects to broker.
SST: init_SST().
int connect_to_netdrv(netdrv_t* drv);
TCP: connect()
MQTT: Federate publishes to “{Federation_ID}_RTI”, then subscribes to ”{Federation_ID}_RTI_to_fed{fed_id}”
SST: Requests session key from Auth, and request connection to RTI or federate server.
Major changes
MQTT
SST
Netdriver supports SST.
RTI must be built with
-DCOMM_TYPE=SST.Instructions
OpenSSL 3.0 above must be installed.
Auth.Java 11 or above must be installed.
The path must be in the same directory with
lingua-francafor the script to work. This is for key paths, and auto generated.configfiles.-DCOMM_TYPE=SST.comm-type:SSTto.lfcode.Input for configuration is required.
Each federate binaries and the RTI needs a config as an input with the
-sstor--sstoption.The code generator copies the skeleton config from
reactor-c/core/federated/network/SSTskeleton.config.It writes three components.
The generated federate's sst config will be saved such as
test/C/fed-gen/SimpleFederated/src-gen/federate__fed1/core/federated/network/federate__fed1.configThe RTI's config will be saved such as
test/C/fed-gen/SimpleFederated/src-gen/rti.config. However, this RTI config is only for local environments.Refactoring
federate.c
lf_create_server():Separate the part sending the address_advertisement message to the RTI. The implementation is in each lf_STACK_support.c. In TCP connections, it will send the port as it did. In MQTT connections, it will do nothing, because it does not need the RTI to connect to another federate. (The MQTT will do a handshake using the federateID, which is already known).
lf_connect_to_federate():This is a very large function having three parts.
1-1. Send the RTI an ADDRESS_QUERY message, sending the remote federate's ID which it wants to connect to.
1-2. Receive ADDRESS_QUERY_REPLY message from RTI, getting the port number and IP address of the requested federate.
2. Connect to the remote federate, using the port and IP address.
3-1. Send my federateID and federationID
3-2. Receive an ACK.
I splitted this into three parts, using helper functions. However, part 1, sending address query messages, will have different implementations depending on the network communication type. So, it is defined in netdriver.h, and implemented in lf_STACK_support.c
Maximum length of Federation ID is needed.
Change response message of RTI to federate MSG_TYPE_ADDRESS_QUERY
** THIS IS ALREADY FIXED IN #392
The federate first sends MSG_TYPE_ADDRESS_QUERY with total 5bytes (1 Header + 2 federate id).
Then, the RTI responds with the same header MSG_TYPE_ADDRESS_QUERY with (1 Header + 4 port_num + 4 IP address).
Due to my approach needing unique message headers, the response of the RTI will be MSG_TYPE_ADDRESS_QUERY_REPLY.