diff --git a/README.md b/README.md index bcb8432f..03787836 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ Full source documentation can be found at https://derecho-project.github.io/. # Derecho [![Build Status](https://travis-ci.com/Derecho-Project/derecho.svg?branch=master)](https://travis-ci.com/Derecho-Project/derecho) -This is the main repository for the Derecho project. It unifies the RDMC, SST, and Derecho modules under a single, easy-to-use repository. +This is the main repository for the Derecho project. It unifies the RDMC, SST, and Derecho modules under a single, easy-to-use repository. ## Intended use cases and assumptions. Derecho is aimed at supporting what are called "cloud micro-services", meaning pools of servers that would reside in a cluster or on a cloud data-center, probably to support the "first tier" where requests arrive from external clients, and that perform some kind of well-defined subtask like data cleaning, image classification, compression and deduplication, etc. Although the system could definitely run in other settings, if you stray too far from our intended use cases, you'll probably trigger timeout-related crashes that we might not be very eager to try and "fix". We recommend using Zookeeper or some other tool if you are aiming at a very different setup. @@ -27,7 +27,7 @@ Derecho is a library that helps you build replicated, fault-tolerant services in * Linux (other operating systems don't currently support the RDMA features we use) * A C++ compiler supporting C++17: GCC 7.3+ or Clang 7+ * CMake 2.8.1 or newer -* The SSL/TLS Library. On Ubuntu and other Debian-like systems, you can install package `libssl-dev`. We tested with v1.0.2n. But it should work for any version >= 1.0 +* The SSL/TLS Library. On Ubuntu and other Debian-like systems, you can install package `libssl-dev`. We tested with v1.0.2n. But it should work for any version >= 1.0 * The "rdmacm" and "ibverbs" system libraries for Linux, at version 17.1 or higher. On Ubuntu and other Debian-like systems, these are in the packages `librdmacm-dev` and `libibverbs-dev`. * [`spdlog`](https://github.com/gabime/spdlog), a logging library, v1.3.1 or newer. On Ubuntu 19.04 and later this can be installed with the package `libspdlog-dev`. The version of spdlog in Ubuntu 18.04's repositories is too old, but if you are running Ubuntu 18.04 you can download the `libspdlog-dev` package [here](https://packages.ubuntu.com/disco/libspdlog-dev) and install it manually with no other dependencies needed. * The Open Fabric Interface (OFI) library: [`libfabric`](https://github.com/ofiwg/libfabric). To avoid compatibility issue, please install `v1.7.0` from source code. ([Installation script](https://github.com/Derecho-Project/derecho/blob/master/scripts/prerequisites/install-libfabric.sh)) @@ -68,20 +68,20 @@ To uninstall, run: To build your own derecho executable, simple run: * `g++ -std=c++1z -o myapp myapp.cpp -lderecho -pthread` -To use Derecho in your code, you simply need to +To use Derecho in your code, you simply need to - include the header `derecho/core/derecho.hpp` in your \*.h \*.hpp or \*.cpp files, and -- specify a configuration file, either by setting environment variable `DERECHO_CONF_FILE` or by placing a file named `derecho.cfg` in the working directory. A sample configuration file along with an explanation can be found in `/share/derecho/derecho-sample.cfg`. +- specify a configuration file, either by setting environment variable `DERECHO_CONF_FILE` or by placing a file named `derecho.cfg` in the working directory. A sample configuration file along with an explanation can be found in `/share/derecho/derecho-sample.cfg`. -The configuration file consists of three sections: **DERECHO**, **RDMA**, and **PERS**. The **DERECHO** section includes core configuration options for a Derecho instance, which every application will need to customize. The **RDMA** section includes options for RDMA hardware specifications. The **PERS** section allows you to customize the persistent layer's behavior. +The configuration file consists of three sections: **DERECHO**, **RDMA**, and **PERS**. The **DERECHO** section includes core configuration options for a Derecho instance, which every application will need to customize. The **RDMA** section includes options for RDMA hardware specifications. The **PERS** section allows you to customize the persistent layer's behavior. #### Configuring Core Derecho -Applications need to tell the Derecho library which node is the initial leader with the options **leader_ip** and **leader_gms_port**. Each node then specifies its own ID (**local_id**) and the IP address and ports it will use for Derecho component services (**local_ip**, **gms_port**, **rpc_port**, **sst_port**, and **rdmc_port**). Also, if using external clients, applications need to specify the ports serving external clients (**leader_external_port** and **external_port**); +Applications need to tell the Derecho library which node is the initial leader with the options **leader_ip** and **leader_gms_port**. Each node then specifies its own ID (**local_id**) and the IP address and ports it will use for Derecho component services (**local_ip**, **gms_port**, **state_transfer_port**, **sst_port**, and **rdmc_port**). Also, if using external clients, applications need to specify the ports serving external clients (**leader_external_port** and **external_port**); The other important parameters are the message sizes. Since Derecho pre-allocates buffers for RDMA communication, each application should decide on an optimal buffer size based on the amount of data it expects to send at once. If the buffer size is much larger than the messages an application actually sends, Derecho will pin a lot of memory and leave it underutilized. If the buffer size is smaller than the application's actual message size, it will have to split messages into segments before sending them, causing unnecessary overhead. Three message-size options control the memory footprint and performance of Derecho. In all cases, larger values will increase the memory (DRAM) footprint of the application, and it is fairly easy to end up with a huge memory size if you just pick giant values. The defaults keep the memory size smaller, but can reduce performance if an application is sending high rates of larger messages. -The options are named **max_payload_size**, **max_smc_payload_size**, **block_size**, **max_p2p_request_payload_size**, and **max_p2p_reply_payload_size**. +The options are named **max_payload_size**, **max_smc_payload_size**, **block_size**, **max_p2p_request_payload_size**, and **max_p2p_reply_payload_size**. No message bigger than **max_payload_size** will be sent by Derecho multicast(`Replicated<>::send()`). No message bigger than **max_p2p_request_payload_size** will be sent by Derecho p2p send(`Replicated<>::p2p_send()` or `ExternalClientCaller<>::p2p_send()`). No reply bigger than **max_p2p_reply_payload_size** will be sent to carry the return values any multicast or p2p send. @@ -132,7 +132,7 @@ We also allow applications to specify configuration options on the command line. ```cpp #define NUM_OF_APP_ARGS () // specify the number of application arguments. int main(int argc, char* argv[]) { - if((argc < (NUM_OF_APP_ARGS+1)) || + if((argc < (NUM_OF_APP_ARGS+1)) || ((argc > (NUM_OF_APP_ARGS+1)) && strcmp("--", argv[argc - NUM_OF_APP_ARGS - 1]))) { cout << "Invalid command line arguments." << endl; cout << "USAGE:" << argv[0] << "[ derecho-config-list -- ] application-argument-list" << endl; @@ -185,7 +185,7 @@ The file `simple_replicated_objects.cpp` within applications/demos shows a compl ### Replicated Objects One of the core building blocks of Derecho is the concept of a Replicated Object. This provides a simple way for you to define state that is replicated among several machines and a set of RPC functions that operate on that state. -A Replicated Object is any class that (1) is serializable with the mutils-serialization framework and (2) implements a static method called `register_functions()`. The [mutils-serialization](https://github.com/mpmilano/mutils-serialization) library should have more documentation on making objects serializable, but the most straightforward way is to inherit `mutils::ByteRepresentable`, use the macro `DEFAULT_SERIALIZATION_SUPPORT`, and write an element-by-element constructor. The `register_functions()` method is how your class specifies to Derecho which of its methods should be converted to RPC functions and what their numeric "function tags" should be. It should return a `std::tuple` containing a pointer to each RPC-callable method, wrapped in the template function `derecho::rpc::tag`, whose template parameter is an integer constant. We have provided a default implementation of this function with the macro `REGISTER_RPC_FUNCTIONS`, which registers each method in its argument using the integer constant generated by the macro `RPC_NAME`. Here is an example of a Replicated Object declaration that uses the default implementation macros: +A Replicated Object is any class that (1) is serializable with the mutils-serialization framework and (2) implements a static method called `register_functions()`. The [mutils-serialization](https://github.com/mpmilano/mutils-serialization) library should have more documentation on making objects serializable, but the most straightforward way is to inherit `mutils::ByteRepresentable`, use the macro `DEFAULT_SERIALIZATION_SUPPORT`, and write an element-by-element constructor. The `register_functions()` method is how your class specifies to Derecho which of its methods should be converted to RPC functions and what their numeric "function tags" should be. It should return a `std::tuple` containing a pointer to each RPC-callable method, wrapped in the template function `derecho::rpc::tag`, whose template parameter is an integer constant. We have provided a default implementation of this function with the macro `REGISTER_RPC_FUNCTIONS`, which registers each method in its argument using the integer constant generated by the macro `RPC_NAME`. Here is an example of a Replicated Object declaration that uses the default implementation macros: ```cpp class Cache : public mutils::ByteRepresentable { @@ -193,7 +193,7 @@ class Cache : public mutils::ByteRepresentable { public: void put(const std::string& key, const std::string& value); - std::string get(const std::string& key); + std::string get(const std::string& key); bool contains(const std::string& key); bool invalidate(const std::string& key); Cache() : cache_map() {} @@ -207,9 +207,9 @@ This object has one field, `cache_map`, so the DEFAULT\_SERIALIZATION\_SUPPORT m ### Groups and Subgroups -Derecho organizes nodes (machines or processes in a system) into Groups, which can then be divided into subgroups and shards. Any member of a Group can communicate with any other member, and all run the same group-management service that handles failures and accepts new members. Subgroups, which are any subset of the nodes in a Group, correspond to Replicated Objects; each subgroup replicates the state of a Replicated Object and any member of the subgroup can handle RPC calls on that object. Shards are disjoint subsets of a subgroup that each maintain their own state, so one subgroup can replicate multiple instances of the same type of Replicated Object. A Group must be statically configured with the types of Replicated Objects it can support, but the number of subgroups and their exact membership can change at runtime according to functions that you provide. +Derecho organizes nodes (machines or processes in a system) into Groups, which can then be divided into subgroups and shards. Any member of a Group can communicate with any other member, and all run the same group-management service that handles failures and accepts new members. Subgroups, which are any subset of the nodes in a Group, correspond to Replicated Objects; each subgroup replicates the state of a Replicated Object and any member of the subgroup can handle RPC calls on that object. Shards are disjoint subsets of a subgroup that each maintain their own state, so one subgroup can replicate multiple instances of the same type of Replicated Object. A Group must be statically configured with the types of Replicated Objects it can support, but the number of subgroups and their exact membership can change at runtime according to functions that you provide. -Note that more than one subgroup can use the same type of Replicated Object, so there can be multiple independent instances of a Replicated Object in a Group even if those subgroups are not sharded. A subgroup is usually identified by the type of Replicated Object it implements and an integral index number specifying which subgroup of that type it is. +Note that more than one subgroup can use the same type of Replicated Object, so there can be multiple independent instances of a Replicated Object in a Group even if those subgroups are not sharded. A subgroup is usually identified by the type of Replicated Object it implements and an integral index number specifying which subgroup of that type it is. To start using Derecho, a process must either start or join a Group by constructing an instance of `derecho::Group`, which then provides the interface for interacting with other nodes in the Group. (The difference between starting and joining a group is simply a matter of calling a different constructor). A `derecho::Group` expects a set of variadic template parameters representing the types of Replicated Objects that it can support in its subgroups. For example, this declaration is a pointer to a Group object that can have subgroups of type LoadBalancer, Cache, and Storage: @@ -268,7 +268,7 @@ For all three types of Replicated Object, the function creates one subgroup and Although the subgroup allocation function is the most important part of constructing a `derecho::Group`, it requires a few additional parameters. * A set of **callback functions** that will be notified when each Derecho message is delivered to the node (stability callback) or persisted to disk (persistence callback). These can be null, and are probably not useful if you're only using the Replicated Object features of Derecho (since the "messages" will be serialized RPC calls). * A set of **View upcalls** that will be notified when the group experiences a View change event (nodes fail or join the group). This is optional and can be empty, but it can be useful for adding additional failure-handling or load-balancing behavior to your application. -* For each template parameter in the type of `derecho::Group`, its constructor will expect an additional argument of type `derecho::Factory`, which is a function or functor that constructs instances of the Replicated Object (it's just an alias for `std::function(void)>`). +* For each template parameter in the type of `derecho::Group`, its constructor will expect an additional argument of type `derecho::Factory`, which is a function or functor that constructs instances of the Replicated Object (it's just an alias for `std::function(void)>`). ### Invoking RPC Functions @@ -335,7 +335,7 @@ class PFoo : public mutils::ByteRepresentable { public: virtual ~PFoo() noexcept (true) {} int read_state() { - return *pint; + return *pint; } bool change_state(int new_int) { if(new_int == *pint) { @@ -345,7 +345,7 @@ public: *pint = new_int; return true; } - + // constructor with PersistentRegistry PFoo(PersistentRegistry * pr) : pint(nullptr,pr) {} PFoo(Persistent & init_pint) : pint(std::move(init_pint)) {} @@ -353,7 +353,7 @@ public: REGISTER_RPC_FUNCTIONS(PFoo, read_state, change_state); }; ``` - + For simplicity, the versioned type is int in this example. You set it up in the same way as a non-versioned member of a replicated object, except that you need to pass the PersistentRegistry from the constructor of the replicated object to the constructor of the `Persistent`. Derecho uses PersistentRegistry to keep track of all the Persistent objects in a single Replicated Object so that it can create versions on updates. The Persistent constructor registers itself in the registry. By default, the Persistent stores its log in the file-system (in a folder called .plog in the current directory). Application can specify memory as the storage location by setting the second template parameter: `Persistent` (or `Volatile` as syntactic sugar). We are working on more store storage types including NVM. @@ -371,7 +371,7 @@ Additionally, it is important for you as the developer to realize that launching Even on HPC systems, which can support MPI at that scale, because MPI doesn't use an all-to-all connection pattern, we have seen these kinds of difficulties at massive scale. In MPI there is one leader and N-1 followers, so the primary pattern that arises is really 1-N connections (more accurately, they do have some cases at runtime (like AllReduce) that can create K x K patterns. I don't know how much success those folks have had with K>=1000, though. My impression is that All Reduce normally runs on a significantly smaller scale. Other KxK situations on MPI are probably delicate to initialize, too.) -At Cornell, up to now, our largest experiments involved cases where we benchmarked RDMC (not the full Derecho) on 1000's of nodes at the LLNL supercomputer center. And it was a nightmare getting to the point where that worked. In the end, we actually had a special batch script to launch them 50 at a time, and have them connect in batches, to avoid overloading the file system and TCP layer. +At Cornell, up to now, our largest experiments involved cases where we benchmarked RDMC (not the full Derecho) on 1000's of nodes at the LLNL supercomputer center. And it was a nightmare getting to the point where that worked. In the end, we actually had a special batch script to launch them 50 at a time, and have them connect in batches, to avoid overloading the file system and TCP layer. Our largest Derecho experiments have been on a Texas supercomputer, where we had successful and completely stable runs on 256 physical nodes and probably could have pushed towards 1024 or more had we not run out of credits: "renting" 1000's of non-virtualized nodes is expensive. Then just as we applied for more credit, they decommissioned the entire machine (Stampede-1). So that whole line of experiments ended abruptly. Still, we do think it could have been carried quite a bit further. In this mode, we felt we were experimenting on a use case and deployment of a kind that Derecho needs to support. diff --git a/include/derecho/conf/conf.hpp b/include/derecho/conf/conf.hpp index 4671e4b6..16113afb 100644 --- a/include/derecho/conf/conf.hpp +++ b/include/derecho/conf/conf.hpp @@ -26,14 +26,14 @@ class Conf { #define CONF_DERECHO_LOCAL_ID "DERECHO/local_id" #define CONF_DERECHO_LOCAL_IP "DERECHO/local_ip" #define CONF_DERECHO_GMS_PORT "DERECHO/gms_port" -#define CONF_DERECHO_RPC_PORT "DERECHO/rpc_port" +#define CONF_DERECHO_STATE_TRANSFER_PORT "DERECHO/state_transfer_port" #define CONF_DERECHO_SST_PORT "DERECHO/sst_port" #define CONF_DERECHO_RDMC_PORT "DERECHO/rdmc_port" #define CONF_DERECHO_EXTERNAL_PORT "DERECHO/external_port" #define CONF_DERECHO_HEARTBEAT_MS "DERECHO/heartbeat_ms" #define CONF_DERECHO_SST_POLL_CQ_TIMEOUT_MS "DERECHO/sst_poll_cq_timeout_ms" #define CONF_DERECHO_RESTART_TIMEOUT_MS "DERECHO/restart_timeout_ms" -#define CONF_DERECHO_ENABLE_BACKUP_RESTART_LEADERS "DERECHO/enable_backup_restart_leaders" +#define CONF_DERECHO_ENABLE_BACKUP_RESTART_LEADERS "DERECHO/enable_backup_restart_leaders" #define CONF_DERECHO_DISABLE_PARTITIONING_SAFETY "DERECHO/disable_partitioning_safety" #define CONF_DERECHO_MAX_P2P_REQUEST_PAYLOAD_SIZE "DERECHO/max_p2p_request_payload_size" @@ -70,7 +70,7 @@ class Conf { {CONF_DERECHO_LOCAL_ID, "0"}, {CONF_DERECHO_LOCAL_IP, "127.0.0.1"}, {CONF_DERECHO_GMS_PORT, "23580"}, - {CONF_DERECHO_RPC_PORT, "28366"}, + {CONF_DERECHO_STATE_TRANSFER_PORT, "28366"}, {CONF_DERECHO_SST_PORT, "37683"}, {CONF_DERECHO_RDMC_PORT, "31675"}, {CONF_DERECHO_EXTERNAL_PORT, "32645"}, @@ -217,8 +217,8 @@ const bool getConfBoolean(const std::string& key); const bool hasCustomizedConfKey(const std::string& key); /** - * Splits a string into a vector of strings using a delimiting string. This is - * helpful for parsing "list-like" config options, which are comma-delimited + * Splits a string into a vector of strings using a delimiting string. This is + * helpful for parsing "list-like" config options, which are comma-delimited * sequences of strings or numbers (so the default delimiter is ","). * @param str The string to split * @param delimiter The string to use as the delimiter for splitting diff --git a/include/derecho/core/detail/derecho_sst.hpp b/include/derecho/core/detail/derecho_sst.hpp index 5befc575..3d3db7b3 100644 --- a/include/derecho/core/detail/derecho_sst.hpp +++ b/include/derecho/core/detail/derecho_sst.hpp @@ -95,7 +95,7 @@ class DerechoSST : public sst::SST { SSTFieldVector joiner_ips; /** joiner_xxx_ports are the port numbers for the joining nodes. */ SSTFieldVector joiner_gms_ports; - SSTFieldVector joiner_rpc_ports; + SSTFieldVector joiner_state_transfer_ports; SSTFieldVector joiner_sst_ports; SSTFieldVector joiner_rdmc_ports; SSTFieldVector joiner_external_ports; @@ -162,7 +162,7 @@ class DerechoSST : public sst::SST { changes(100 + parameters.members.size()), //The extra 100 entries allows for more joins at startup, when the group is very small joiner_ips(100 + parameters.members.size()), joiner_gms_ports(100 + parameters.members.size()), - joiner_rpc_ports(100 + parameters.members.size()), + joiner_state_transfer_ports(100 + parameters.members.size()), joiner_sst_ports(100 + parameters.members.size()), joiner_rdmc_ports(100 + parameters.members.size()), joiner_external_ports(100 + parameters.members.size()), @@ -174,7 +174,7 @@ class DerechoSST : public sst::SST { local_stability_frontier(num_subgroups) { SSTInit(seq_num, delivered_num, persisted_num, vid, suspected, changes, joiner_ips, - joiner_gms_ports, joiner_rpc_ports, joiner_sst_ports, joiner_rdmc_ports, joiner_external_ports, + joiner_gms_ports, joiner_state_transfer_ports, joiner_sst_ports, joiner_rdmc_ports, joiner_external_ports, num_changes, num_committed, num_acked, num_installed, num_received, wedged, global_min, global_min_ready, slots, num_received_sst, local_stability_frontier, rip); @@ -197,7 +197,7 @@ class DerechoSST : public sst::SST { } memset(const_cast(joiner_ips[row]), 0, joiner_ips.size()); memset(const_cast(joiner_gms_ports[row]), 0, joiner_gms_ports.size()); - memset(const_cast(joiner_rpc_ports[row]), 0, joiner_rpc_ports.size()); + memset(const_cast(joiner_state_transfer_ports[row]), 0, joiner_state_transfer_ports.size()); memset(const_cast(joiner_sst_ports[row]), 0, joiner_sst_ports.size()); memset(const_cast(joiner_rdmc_ports[row]), 0, joiner_rdmc_ports.size()); memset(const_cast(joiner_external_ports[row]), 0, joiner_external_ports.size()); diff --git a/include/derecho/core/detail/multicast_group.hpp b/include/derecho/core/detail/multicast_group.hpp index dbc93c4d..c721e32e 100644 --- a/include/derecho/core/detail/multicast_group.hpp +++ b/include/derecho/core/detail/multicast_group.hpp @@ -67,7 +67,7 @@ struct DerechoParams : public mutils::ByteRepresentable { unsigned int window_size; unsigned int heartbeat_ms; rdmc::send_algorithm rdmc_send_algorithm; - uint32_t rpc_port; + uint32_t state_transfer_port; static uint64_t compute_max_msg_size( const uint64_t max_payload_size, @@ -103,14 +103,14 @@ struct DerechoParams : public mutils::ByteRepresentable { unsigned int window_size, unsigned int heartbeat_ms, rdmc::send_algorithm rdmc_send_algorithm, - uint32_t rpc_port) + uint32_t state_transfer_port) : max_reply_msg_size(max_reply_payload_size + sizeof(header)), sst_max_msg_size(max_smc_payload_size + sizeof(header)), block_size(block_size), window_size(window_size), heartbeat_ms(heartbeat_ms), rdmc_send_algorithm(rdmc_send_algorithm), - rpc_port(rpc_port) { + state_transfer_port(state_transfer_port) { //if this is initialized above, DerechoParams turns abstract. idk why. max_msg_size = compute_max_msg_size(max_payload_size, block_size, max_payload_size > max_smc_payload_size); @@ -129,7 +129,7 @@ struct DerechoParams : public mutils::ByteRepresentable { std::string prefix = "SUBGROUP/" + profile + "/"; for(auto& field : Conf::subgroupProfileFields) { if(!hasCustomizedConfKey(prefix + field)) { - std::cout << "key" << (prefix + field) + std::cout << "key" << (prefix + field) << " not found in SUBGROUP section of derecho conf. " " Look at derecho-sample.cfg for more information." << std::endl; @@ -138,29 +138,29 @@ struct DerechoParams : public mutils::ByteRepresentable { } uint64_t max_payload_size = getConfUInt64(prefix + Conf::subgroupProfileFields[0]); - uint64_t max_reply_payload_size = getConfUInt64(prefix + Conf::subgroupProfileFields[1]); + uint64_t max_reply_payload_size = getConfUInt64(prefix + Conf::subgroupProfileFields[1]); uint64_t max_smc_payload_size = getConfUInt64(prefix + Conf::subgroupProfileFields[2]); uint64_t block_size = getConfUInt64(prefix + Conf::subgroupProfileFields[3]); uint32_t window_size = getConfUInt32(prefix + Conf::subgroupProfileFields[4]); uint32_t timeout_ms = getConfUInt32(CONF_DERECHO_HEARTBEAT_MS); const std::string& algorithm = getConfString(prefix + Conf::subgroupProfileFields[5]); - uint32_t rpc_port = getConfUInt32(CONF_DERECHO_RPC_PORT); + uint32_t state_transfer_port = getConfUInt32(CONF_DERECHO_STATE_TRANSFER_PORT); return DerechoParams{ max_payload_size, - max_reply_payload_size, + max_reply_payload_size, max_smc_payload_size, block_size, window_size, timeout_ms, DerechoParams::send_algorithm_from_string(algorithm), - rpc_port, + state_transfer_port, }; } DEFAULT_SERIALIZATION_SUPPORT(DerechoParams, max_msg_size, max_reply_msg_size, sst_max_msg_size, block_size, window_size, - heartbeat_ms, rdmc_send_algorithm, rpc_port); + heartbeat_ms, rdmc_send_algorithm, state_transfer_port); }; /** @@ -368,7 +368,7 @@ class MulticastGroup { * @param msg_ts The timestamp of the message */ void deliver_message(RDMCMessage& msg, const subgroup_id_t& subgroup_num, - const persistent::version_t& version, const uint64_t& msg_timestamp); + const persistent::version_t& version, const uint64_t& msg_timestamp); /** * Same as the other deliver_message, but for the SSTMessage type @@ -378,7 +378,7 @@ class MulticastGroup { * @param msg_ts The timestamp of this message */ void deliver_message(SSTMessage& msg, const subgroup_id_t& subgroup_num, - const persistent::version_t& version, const uint64_t& msg_timestamp); + const persistent::version_t& version, const uint64_t& msg_timestamp); /** * Enqueues a single message for persistence with the persistence manager. @@ -393,7 +393,7 @@ class MulticastGroup { * false if the message is a null message */ bool version_message(RDMCMessage& msg, const subgroup_id_t& subgroup_num, - const persistent::version_t& version, const uint64_t& msg_timestamp); + const persistent::version_t& version, const uint64_t& msg_timestamp); /** * Same as the other version_message, but for the SSTMessage type. * @param msg The message that should cause a new version to be registered @@ -405,7 +405,7 @@ class MulticastGroup { * false if the message is a null message */ bool version_message(SSTMessage& msg, const subgroup_id_t& subgroup_num, - const persistent::version_t& version, const uint64_t& msg_timestamp); + const persistent::version_t& version, const uint64_t& msg_timestamp); uint32_t get_num_senders(const std::vector& shard_senders) { uint32_t num = 0; diff --git a/include/derecho/core/detail/view_manager.hpp b/include/derecho/core/detail/view_manager.hpp index a9808b06..4f4647a1 100644 --- a/include/derecho/core/detail/view_manager.hpp +++ b/include/derecho/core/detail/view_manager.hpp @@ -191,13 +191,13 @@ class ViewManager { * Contains a TCP connection to each member of the group, for the purpose * of transferring new Views and state information (serialized Replicated * Objects) to new members during a view change. Each socket is connected - * to the (badly-named) RPC port of the corresponding member. + * to the transfer port of the corresponding member. */ tcp::tcp_connections tcp_sockets; - /** + /** * The socket that made the initial connection to the restart leader, if this - * node is a non-leader. This is only used during the initial startup phase; + * node is a non-leader. This is only used during the initial startup phase; * after the Group constructor finishes and start() is called, it will be null. */ std::unique_ptr leader_connection; @@ -257,7 +257,7 @@ class ViewManager { /** * On a graceful exit, nodes will be agree to leave at some point, where - * the view manager should stop throw exception on "failure". Set + * the view manager should stop throw exception on "failure". Set * 'bSilence' to keep the view manager calm on detecting intended node * "failure." */ @@ -483,7 +483,7 @@ class ViewManager { /* ---------------------------------------------------------------------------------- */ /* ------------------------ Setup/constructor helpers ------------------------------- */ - /** + /** * The initial start-up procedure (basically a constructor) for the case * where there is no logged state on disk and the group is doing a "fresh * start." At the end of this function this node has constructed or received @@ -500,7 +500,7 @@ class ViewManager { /** Constructor helper for the leader when it first starts; waits for enough * new nodes to join to make the first view adequately provisioned. */ void await_first_view(); - /** + /** * Constructor helper for non-leader nodes; encapsulates receiving and * deserializing a View, DerechoParams, and state-transfer leaders (old * shard leaders) from the leader. @@ -514,7 +514,7 @@ class ViewManager { void initialize_rdmc_sst(); /** * Helper for joining an existing group; receives the View and parameters from the leader. - * @return true if the leader successfully sent the View, false if the leader crashed + * @return true if the leader successfully sent the View, false if the leader crashed * (i.e. a socket operation to it failed) before completing the process */ bool receive_initial_view(); @@ -640,7 +640,7 @@ class ViewManager { * Constructor for either the leader or non-leader of a group. * @param subgroup_info The set of functions defining subgroup membership * for this group. - * @param subgroup_type_order A vector of type_index in the same order as + * @param subgroup_type_order A vector of type_index in the same order as * the template parameters to the Group class * @param any_persistent_objects True if any of the subgroup types in this * group use Persistent fields, false otherwise diff --git a/include/derecho/core/view.hpp b/include/derecho/core/view.hpp index 9f27e30f..35f0dfbb 100644 --- a/include/derecho/core/view.hpp +++ b/include/derecho/core/view.hpp @@ -24,7 +24,7 @@ namespace derecho { * values corresponds to a "port" field in the IpAndPorts struct. */ enum class PortType { GMS, //!< GMS - RPC, //!< RPC + TRANSFER, //!< TRANSFER SST, //!< SST RDMC, //!< RDMC EXTERNAL }; //!< EXTERNAL @@ -35,7 +35,7 @@ enum class PortType { GMS, //!< GMS struct IpAndPorts : public mutils::ByteRepresentable { ip_addr_t ip_address; uint16_t gms_port; - uint16_t rpc_port; + uint16_t state_transfer_port; uint16_t sst_port; uint16_t rdmc_port; uint16_t external_port; @@ -47,28 +47,28 @@ struct IpAndPorts : public mutils::ByteRepresentable { */ IpAndPorts(const ip_addr_t& ip_address, const uint16_t gms_port, - const uint16_t rpc_port, + const uint16_t state_transfer_port, const uint16_t sst_port, const uint16_t rdmc_port, const uint16_t external_port) : ip_address(ip_address), gms_port(gms_port), - rpc_port(rpc_port), + state_transfer_port(state_transfer_port), sst_port(sst_port), rdmc_port(rdmc_port), external_port(external_port) {} - IpAndPorts() : ip_address{}, gms_port(0), rpc_port(0), sst_port(0), rdmc_port(0), external_port(0) {} + IpAndPorts() : ip_address{}, gms_port(0), state_transfer_port(0), sst_port(0), rdmc_port(0), external_port(0) {} - DEFAULT_SERIALIZATION_SUPPORT(IpAndPorts, ip_address, gms_port, rpc_port, + DEFAULT_SERIALIZATION_SUPPORT(IpAndPorts, ip_address, gms_port, state_transfer_port, sst_port, rdmc_port, external_port); inline bool operator==(const IpAndPorts& o) const { return std::tie(ip_address, - gms_port, rpc_port, + gms_port, state_transfer_port, sst_port, rdmc_port, external_port) == std::tie(o.ip_address, - o.gms_port, o.rpc_port, + o.gms_port, o.state_transfer_port, o.sst_port, o.rdmc_port, o.external_port); } @@ -182,7 +182,7 @@ class View : public mutils::ByteRepresentable { * position in this vector. */ std::vector subgroup_type_order; /** Maps the (type, index) pairs used by users to identify subgroups to the - * internal subgroup IDs generated by ViewManager during SST setup. + * internal subgroup IDs generated by ViewManager during SST setup. * The order of ids in the vector follows the order in which the user created * those subgroups of the same type. */ diff --git a/scripts/travis-ci/derecho0.cfg b/scripts/travis-ci/derecho0.cfg index 63fda0fa..812d1dfe 100644 --- a/scripts/travis-ci/derecho0.cfg +++ b/scripts/travis-ci/derecho0.cfg @@ -11,8 +11,8 @@ local_id = 0 local_ip = 127.0.0.1 # derecho gms port gms_port = 23580 -# derecho rpc port -rpc_port = 28366 +# derecho state-transfer port +state_transfer_port = 28366 # sst tcp port sst_port = 37683 # rdmc tcp port @@ -76,7 +76,7 @@ provider = sockets # For verbs provider, domain is the device name (ibv_devices) domain = lo -# 3. tx_depth +# 3. tx_depth # tx_depth applies to hints->tx_attr->size, where hint is a struct fi_info object. # see https://ofiwg.github.io/libfabric/master/man/fi_getinfo.3.html tx_depth = 256 diff --git a/scripts/travis-ci/derecho1.cfg b/scripts/travis-ci/derecho1.cfg index 9568aafd..c1a0cb6c 100644 --- a/scripts/travis-ci/derecho1.cfg +++ b/scripts/travis-ci/derecho1.cfg @@ -11,8 +11,8 @@ local_id = 1 local_ip = 127.0.0.1 # derecho gms port gms_port = 33580 -# derecho rpc port -rpc_port = 38366 +# derecho state-transfer port +state_transfer_port = 38366 # sst tcp port sst_port = 47683 # rdmc tcp port @@ -76,7 +76,7 @@ provider = sockets # For verbs provider, domain is the device name (ibv_devices) domain = lo -# 3. tx_depth +# 3. tx_depth # tx_depth applies to hints->tx_attr->size, where hint is a struct fi_info object. # see https://ofiwg.github.io/libfabric/master/man/fi_getinfo.3.html tx_depth = 256 diff --git a/src/conf/conf.cpp b/src/conf/conf.cpp index 5c8052aa..ee6a878b 100644 --- a/src/conf/conf.cpp +++ b/src/conf/conf.cpp @@ -38,7 +38,7 @@ struct option Conf::long_options[] = { MAKE_LONG_OPT_ENTRY(CONF_DERECHO_LOCAL_ID), MAKE_LONG_OPT_ENTRY(CONF_DERECHO_LOCAL_IP), MAKE_LONG_OPT_ENTRY(CONF_DERECHO_GMS_PORT), - MAKE_LONG_OPT_ENTRY(CONF_DERECHO_RPC_PORT), + MAKE_LONG_OPT_ENTRY(CONF_DERECHO_STATE_TRANSFER_PORT), MAKE_LONG_OPT_ENTRY(CONF_DERECHO_SST_PORT), MAKE_LONG_OPT_ENTRY(CONF_DERECHO_RDMC_PORT), MAKE_LONG_OPT_ENTRY(CONF_DERECHO_EXTERNAL_PORT), @@ -159,11 +159,11 @@ const bool hasCustomizedConfKey(const std::string& key) { std::vector split_string(const std::string& str, const std::string& delimiter) { std::vector result; - std::size_t lastpos = 0; - std::size_t nextpos = 0; + std::size_t lastpos = 0; + std::size_t nextpos = 0; while((nextpos = str.find(delimiter, lastpos)) != std::string::npos) { result.emplace_back(str.substr(lastpos, nextpos)); - lastpos = nextpos + delimiter.length(); + lastpos = nextpos + delimiter.length(); } result.emplace_back(str.substr(lastpos)); return result; diff --git a/src/conf/derecho-sample.cfg b/src/conf/derecho-sample.cfg index 2bd1a6ae..21bccaed 100644 --- a/src/conf/derecho-sample.cfg +++ b/src/conf/derecho-sample.cfg @@ -15,8 +15,8 @@ local_id = 0 local_ip = 127.0.0.1 # derecho gms port gms_port = 23580 -# derecho rpc port -rpc_port = 28366 +# derecho state-transfer port +state_transfer_port = 28366 # sst tcp port sst_port = 37683 # rdmc tcp port @@ -30,14 +30,14 @@ heartbeat_ms = 1 # sst poll completion queue timeout in millisecond sst_poll_cq_timeout_ms = 100 # This is the maximum time a restart leader will wait for other nodes to restart -# before proceeding with the restart if it has a quorum; it's a "grace period" -# that allows more nodes to be included in the restart quorum at the cost of +# before proceeding with the restart if it has a quorum; it's a "grace period" +# that allows more nodes to be included in the restart quorum at the cost of # taking longer to restart. restart_timeout_ms = 2000 # This setting controls the experimental "backup restart leaders" feature. If -# false, only the first leader in the restart_leaders list will be contacted +# false, only the first leader in the restart_leaders list will be contacted # during a restart (the rest are ignored), and the group will fail to restart -# if this leader crashes. If true (enabled), restarting nodes will try +# if this leader crashes. If true (enabled), restarting nodes will try # contacting the backup leaders in order once they detect that the first restart # leader has failed. The default is false since failure detection during restart # is unreliable and may cause a slow restart leader to be treated as failed. @@ -46,7 +46,7 @@ enable_backup_restart_leaders = false # By disabling this feature, the derecho is allowed to run when active # members cannot form a majority. Please be aware of the 'split-brain' # syndrome:https://en.wikipedia.org/wiki/Split-brain and make sure your -# application is fine with it. +# application is fine with it. # To help the user play with derecho at beginning, we disabled the # partitioning safety. We suggest to set it to false for serious deployment disable_partitioning_safety = true @@ -130,7 +130,7 @@ provider = sockets # For verbs provider, domain is the device name (ibv_devices) domain = eth0 -# 3. tx_depth +# 3. tx_depth # tx_depth applies to hints->tx_attr->size, where hint is a struct fi_info object. # see https://ofiwg.github.io/libfabric/master/man/fi_getinfo.3.html tx_depth = 256 diff --git a/src/core/derecho_sst.cpp b/src/core/derecho_sst.cpp index 7be7eb3f..15557887 100644 --- a/src/core/derecho_sst.cpp +++ b/src/core/derecho_sst.cpp @@ -19,9 +19,9 @@ void DerechoSST::init_local_row_from_previous(const DerechoSST& old_sst, const i memcpy(const_cast(joiner_gms_ports[local_row]), const_cast(old_sst.joiner_gms_ports[row] + num_changes_installed), (old_sst.joiner_gms_ports.size() - num_changes_installed) * sizeof(uint16_t)); - memcpy(const_cast(joiner_rpc_ports[local_row]), - const_cast(old_sst.joiner_rpc_ports[row] + num_changes_installed), - (old_sst.joiner_rpc_ports.size() - num_changes_installed) * sizeof(uint16_t)); + memcpy(const_cast(joiner_state_transfer_ports[local_row]), + const_cast(old_sst.joiner_state_transfer_ports[row] + num_changes_installed), + (old_sst.joiner_state_transfer_ports.size() - num_changes_installed) * sizeof(uint16_t)); memcpy(const_cast(joiner_sst_ports[local_row]), const_cast(old_sst.joiner_sst_ports[row] + num_changes_installed), (old_sst.joiner_sst_ports.size() - num_changes_installed) * sizeof(uint16_t)); @@ -60,9 +60,9 @@ void DerechoSST::init_local_change_proposals(const int other_row) { memcpy(const_cast(joiner_gms_ports[local_row]), const_cast(joiner_gms_ports[other_row]), joiner_gms_ports.size() * sizeof(uint16_t)); - memcpy(const_cast(joiner_rpc_ports[local_row]), - const_cast(joiner_rpc_ports[other_row]), - joiner_rpc_ports.size() * sizeof(uint16_t)); + memcpy(const_cast(joiner_state_transfer_ports[local_row]), + const_cast(joiner_state_transfer_ports[other_row]), + joiner_state_transfer_ports.size() * sizeof(uint16_t)); memcpy(const_cast(joiner_sst_ports[local_row]), const_cast(joiner_sst_ports[other_row]), joiner_sst_ports.size() * sizeof(uint16_t)); @@ -107,9 +107,9 @@ std::string DerechoSST::to_string() const { for(int n = 0; n < (num_changes[row] - num_installed[row]); ++n) { s << joiner_gms_ports[row][n] << " "; } - s << "}, joiner_rpc_ports={ "; + s << "}, joiner_state_transfer_ports={ "; for(int n = 0; n < (num_changes[row] - num_installed[row]); ++n) { - s << joiner_rpc_ports[row][n] << " "; + s << joiner_state_transfer_ports[row][n] << " "; } s << "}, joiner_sst_ports={ "; for(int n = 0; n < (num_changes[row] - num_installed[row]); ++n) { diff --git a/src/core/git_version.cpp b/src/core/git_version.cpp index c78700a8..8c05f8de 100644 --- a/src/core/git_version.cpp +++ b/src/core/git_version.cpp @@ -10,11 +10,11 @@ namespace derecho { -const int MAJOR_VERSION = 0; -const int MINOR_VERSION = 9; -const int PATCH_VERSION = 2; -const int COMMITS_AHEAD_OF_VERSION = 142; -const char* VERSION_STRING = "0.9.2"; -const char* VERSION_STRING_PLUS_COMMITS = "0.9.2+142"; +const int MAJOR_VERSION = 1; +const int MINOR_VERSION = 0; +const int PATCH_VERSION = 0; +const int COMMITS_AHEAD_OF_VERSION = 6; +const char* VERSION_STRING = "1.0.0"; +const char* VERSION_STRING_PLUS_COMMITS = "1.0.0+6"; } diff --git a/src/core/restart_state.cpp b/src/core/restart_state.cpp index e4752fbc..d0d1fc4c 100644 --- a/src/core/restart_state.cpp +++ b/src/core/restart_state.cpp @@ -1,5 +1,5 @@ -#include #include +#include #include #include @@ -139,8 +139,8 @@ void RestartLeaderState::await_quorum(tcp::connection_listener& server_socket) { //Receive the joining node's ports - this is part of the standard join logic uint16_t joiner_gms_port = 0; client_socket->read(joiner_gms_port); - uint16_t joiner_rpc_port = 0; - client_socket->read(joiner_rpc_port); + uint16_t joiner_state_transfer_port = 0; + client_socket->read(joiner_state_transfer_port); uint16_t joiner_sst_port = 0; client_socket->read(joiner_sst_port); uint16_t joiner_rdmc_port = 0; @@ -149,7 +149,8 @@ void RestartLeaderState::await_quorum(tcp::connection_listener& server_socket) { client_socket->read(joiner_external_port); const ip_addr_t& joiner_ip = client_socket->get_remote_ip(); rejoined_node_ips_and_ports[join_request.joiner_id] = {joiner_ip, joiner_gms_port, - joiner_rpc_port, joiner_sst_port, joiner_rdmc_port, joiner_external_port}; + joiner_state_transfer_port, joiner_sst_port, + joiner_rdmc_port, joiner_external_port}; //Done receiving from this socket (for now), so store it in waiting_join_sockets for later waiting_join_sockets.emplace(join_request.joiner_id, std::move(*client_socket)); //Check for quorum @@ -413,7 +414,7 @@ std::unique_ptr RestartLeaderState::update_curr_and_next_restart_view() { nodes_to_add_in_next_view.emplace_back(my_id); ips_and_ports_to_add_in_next_view.emplace_back(getConfString(CONF_DERECHO_LOCAL_IP), getConfUInt16(CONF_DERECHO_GMS_PORT), - getConfUInt16(CONF_DERECHO_RPC_PORT), + getConfUInt16(CONF_DERECHO_STATE_TRANSFER_PORT), getConfUInt16(CONF_DERECHO_SST_PORT), getConfUInt16(CONF_DERECHO_RDMC_PORT), getConfUInt16(CONF_DERECHO_EXTERNAL_PORT)); diff --git a/src/core/view_manager.cpp b/src/core/view_manager.cpp index d1f48c7c..dd35de56 100644 --- a/src/core/view_manager.cpp +++ b/src/core/view_manager.cpp @@ -40,7 +40,7 @@ ViewManager::ViewManager( subgroup_type_order(subgroup_type_order), tcp_sockets(getConfUInt32(CONF_DERECHO_LOCAL_ID), {{getConfUInt32(CONF_DERECHO_LOCAL_ID), - {getConfString(CONF_DERECHO_LOCAL_IP), getConfUInt16(CONF_DERECHO_RPC_PORT)}}}), + {getConfString(CONF_DERECHO_LOCAL_IP), getConfUInt16(CONF_DERECHO_STATE_TRANSFER_PORT)}}}), subgroup_objects(object_reference_map), any_persistent_objects(any_persistent_objects), persistence_manager_callbacks(_persistence_manager_callbacks) { @@ -104,7 +104,7 @@ void ViewManager::startup_to_first_view() { std::vector{ {getConfString(CONF_DERECHO_LOCAL_IP), getConfUInt16(CONF_DERECHO_GMS_PORT), - getConfUInt16(CONF_DERECHO_RPC_PORT), + getConfUInt16(CONF_DERECHO_STATE_TRANSFER_PORT), getConfUInt16(CONF_DERECHO_SST_PORT), getConfUInt16(CONF_DERECHO_RDMC_PORT), getConfUInt16(CONF_DERECHO_EXTERNAL_PORT)}}, @@ -269,7 +269,7 @@ bool ViewManager::receive_initial_view() { } try { leader_connection->write(getConfUInt16(CONF_DERECHO_GMS_PORT)); - leader_connection->write(getConfUInt16(CONF_DERECHO_RPC_PORT)); + leader_connection->write(getConfUInt16(CONF_DERECHO_STATE_TRANSFER_PORT)); leader_connection->write(getConfUInt16(CONF_DERECHO_SST_PORT)); leader_connection->write(getConfUInt16(CONF_DERECHO_RDMC_PORT)); leader_connection->write(getConfUInt16(CONF_DERECHO_EXTERNAL_PORT)); @@ -484,7 +484,7 @@ void ViewManager::setup_initial_tcp_connections(const View& initial_view, const if(initial_view.members[i] != my_id) { tcp_sockets.add_node(initial_view.members[i], {initial_view.member_ips_and_ports[i].ip_address, - initial_view.member_ips_and_ports[i].rpc_port}); + initial_view.member_ips_and_ports[i].state_transfer_port}); dbg_default_debug("Established a TCP connection to node {}", initial_view.members[i]); } } @@ -499,7 +499,7 @@ void ViewManager::reinit_tcp_connections(const View& initial_view, const node_id && !tcp_sockets.contains_node(initial_view.members[i])) { tcp_sockets.add_node(initial_view.members[i], {initial_view.member_ips_and_ports[i].ip_address, - initial_view.member_ips_and_ports[i].rpc_port}); + initial_view.member_ips_and_ports[i].state_transfer_port}); dbg_default_debug("Established a TCP connection to node {}", initial_view.members[i]); } } @@ -544,8 +544,8 @@ void ViewManager::await_first_view() { client_socket.write(JoinResponse{JoinResponseCode::OK, my_id}); uint16_t joiner_gms_port = 0; client_socket.read(joiner_gms_port); - uint16_t joiner_rpc_port = 0; - client_socket.read(joiner_rpc_port); + uint16_t joiner_state_transfer_port = 0; + client_socket.read(joiner_state_transfer_port); uint16_t joiner_sst_port = 0; client_socket.read(joiner_sst_port); uint16_t joiner_rdmc_port = 0; @@ -559,7 +559,7 @@ void ViewManager::await_first_view() { curr_view = std::make_unique(curr_view->vid, functional_append(curr_view->members, joiner_id), functional_append(curr_view->member_ips_and_ports, - {joiner_ip, joiner_gms_port, joiner_rpc_port, joiner_sst_port, joiner_rdmc_port, joiner_external_port}), + {joiner_ip, joiner_gms_port, joiner_state_transfer_port, joiner_sst_port, joiner_rdmc_port, joiner_external_port}), std::vector(curr_view->num_members + 1, 0), functional_append(curr_view->joined, joiner_id), std::vector{}, 0, 0, @@ -1084,8 +1084,8 @@ void ViewManager::acknowledge_proposed_change(DerechoSST& gmsSST) { gmsSST.joiner_ips.size()); gmssst::set(gmsSST.joiner_gms_ports[myRank], gmsSST.joiner_gms_ports[leader], gmsSST.joiner_gms_ports.size()); - gmssst::set(gmsSST.joiner_rpc_ports[myRank], gmsSST.joiner_rpc_ports[leader], - gmsSST.joiner_rpc_ports.size()); + gmssst::set(gmsSST.joiner_state_transfer_ports[myRank], gmsSST.joiner_state_transfer_ports[leader], + gmsSST.joiner_state_transfer_ports.size()); gmssst::set(gmsSST.joiner_sst_ports[myRank], gmsSST.joiner_sst_ports[leader], gmsSST.joiner_sst_ports.size()); gmssst::set(gmsSST.joiner_rdmc_ports[myRank], gmsSST.joiner_rdmc_ports[leader], @@ -1586,8 +1586,8 @@ bool ViewManager::receive_join(DerechoSST& gmsSST, const node_id_t joiner_id, tc uint16_t joiner_gms_port = 0; client_socket.read(joiner_gms_port); - uint16_t joiner_rpc_port = 0; - client_socket.read(joiner_rpc_port); + uint16_t joiner_state_transfer_port = 0; + client_socket.read(joiner_state_transfer_port); uint16_t joiner_sst_port = 0; client_socket.read(joiner_sst_port); uint16_t joiner_rdmc_port = 0; @@ -1610,8 +1610,8 @@ bool ViewManager::receive_join(DerechoSST& gmsSST, const node_id_t joiner_id, tc joiner_ip_packed.s_addr); gmssst::set(gmsSST.joiner_gms_ports[curr_view->my_rank][next_change_index], joiner_gms_port); - gmssst::set(gmsSST.joiner_rpc_ports[curr_view->my_rank][next_change_index], - joiner_rpc_port); + gmssst::set(gmsSST.joiner_state_transfer_ports[curr_view->my_rank][next_change_index], + joiner_state_transfer_port); gmssst::set(gmsSST.joiner_sst_ports[curr_view->my_rank][next_change_index], joiner_sst_port); gmssst::set(gmsSST.joiner_rdmc_ports[curr_view->my_rank][next_change_index], @@ -1746,7 +1746,7 @@ void ViewManager::update_tcp_connections(const View& new_view) { for(const node_id_t& joiner_id : new_view.joined) { tcp_sockets.add_node(joiner_id, {new_view.member_ips_and_ports[new_view.rank_of(joiner_id)].ip_address, - new_view.member_ips_and_ports[new_view.rank_of(joiner_id)].rpc_port}); + new_view.member_ips_and_ports[new_view.rank_of(joiner_id)].state_transfer_port}); dbg_default_debug("Established a TCP connection to node {}", joiner_id); } } @@ -1893,10 +1893,10 @@ ViewManager::make_member_ips_and_ports_map(const View& view, const PortType port view.member_ips_and_ports[i].ip_address, view.member_ips_and_ports[i].gms_port}; break; - case PortType::RPC: + case PortType::TRANSFER: member_ips_and_ports_map[view.members[i]] = { view.member_ips_and_ports[i].ip_address, - view.member_ips_and_ports[i].rpc_port}; + view.member_ips_and_ports[i].state_transfer_port}; break; case PortType::SST: member_ips_and_ports_map[view.members[i]] = { @@ -1969,7 +1969,7 @@ std::unique_ptr ViewManager::make_next_view(const std::unique_ptr& c members[new_member_rank] = joiner_id; member_ips_and_ports[new_member_rank] = {joiner_ip, gmsSST.joiner_gms_ports[my_rank][join_index], - gmsSST.joiner_rpc_ports[my_rank][join_index], + gmsSST.joiner_state_transfer_ports[my_rank][join_index], gmsSST.joiner_sst_ports[my_rank][join_index], gmsSST.joiner_rdmc_ports[my_rank][join_index], gmsSST.joiner_external_ports[my_rank][join_index]}; @@ -2179,8 +2179,8 @@ bool ViewManager::copy_prior_leader_proposals(DerechoSST& gmsSST) { gmsSST.joiner_ips.size()); gmssst::set(gmsSST.joiner_gms_ports[my_rank], gmsSST.joiner_gms_ports[longest_changes_rank], gmsSST.joiner_gms_ports.size()); - gmssst::set(gmsSST.joiner_rpc_ports[my_rank], gmsSST.joiner_rpc_ports[longest_changes_rank], - gmsSST.joiner_rpc_ports.size()); + gmssst::set(gmsSST.joiner_state_transfer_ports[my_rank], gmsSST.joiner_state_transfer_ports[longest_changes_rank], + gmsSST.joiner_state_transfer_ports.size()); gmssst::set(gmsSST.joiner_sst_ports[my_rank], gmsSST.joiner_sst_ports[longest_changes_rank], gmsSST.joiner_sst_ports.size()); gmssst::set(gmsSST.joiner_rdmc_ports[my_rank], gmsSST.joiner_rdmc_ports[longest_changes_rank],