-
Notifications
You must be signed in to change notification settings - Fork 1
FAQ
No. See this guide section for more details.
No. UPC++ provides per-operation completions to this end. See this guide section
No. It is a requirement that upcxx::progress() be called for futures to flip to their ready state. No amount of chrono sleeping will flip a future even if the transfer has actually completed.
When I conjoin futures using when_all(), in what order do the operations complete? Is there a possibility of a race condition?
Futures progress sequentially on the calling thread and in unspecified order. UPC++ futures are not a tool for expressing parallelism, but instead non-deterministic sequential behavior. Futures allow a thread to react to events in the order they actually occur, since the order in which communication operations complete is not deterministic. While UPC++ does support multiple threads within each process, and each thread can be managing its own futures, the same future must never be accessed concurrently by multiple threads. Also, UPC++ will never "spawn" new threads. See the section "Futures and Promises" of the UPC++ specification.
The short answer is: NEVER use by-reference lambda captures for RPC, and by-value captures are ONLY safe if the types are TriviallySerializable (TriviallyCopyable). See this guide section for more details.
In general this is NOT guaranteed to work correctly. In particular, systems using Address space layout randomization (ASLR) (a security feature which is enabled by default on many Linux and macOS systems) will load static data and possibly even code segments at different randomized base addresses in virtual memory for each process in the job.
Note your callbacks can safely refer to static/global variables by name, which will be resolved correctly wherever the callback runs. For variables with a dynamic lifetime, the recommended solution is to use a dist_object wrapper; sending a dist_object reference as an RPC argument automatically ensures the callback receives a reference to the corresponding local representative of the dist_object.
Can you make a RPC call from within a RPC callback? When we try to wait on the nested RPC we get an error/hang.
You can absolutely launch RPC, RMA or other communication from within an RPC callback. However you cannot wait on the completion of communication from within the RPC callback. If you run in debug mode you'll get an error message explaining that, e.g.: "You have attempted to wait() on a non-ready future within upcxx progress, this is prohibited because it will never complete."
So if you launch communication from RPC callback context you need to use UPC++'s other completion and asynchrony features to synchronize completion instead of calling future::wait(). There are many different ways to do this, and the best choice depends on the details of what the code is doing. For example, one could request future-based completion on that op and either stash the resulting future in storage outside the callback's stack frame, or better yet schedule a completion callback on the future using future::then() with a continuation of whatever you would have done after wait (more details). There are also operations (like rpc_ff) and completion types (remote_cx::as_rpc) that signal completion at the remote side (more details), so you perform synchronization at the target instead of the initiator.
Does upcxx do any optimization if I happen to call an rpc destined to rank_me()? Do I need to explicitly do anything to make that more efficient?
There is an optimization applied to all targets in the local_team() (shared-memory bypass avoids one payload copy for large payloads using the rendezvous-get protocol), but nothing specific to same-process "loopback" (rank_me).
If you think same-process/loopback RPCs comprise a significant fraction of your RPC traffic, you should probably consider that serialization semantics require copying the argument payload at least once, even for a loopback RPC. Also, the progress semantics require deferring callback invocation until the next user-level progress, so if the input arguments are hot in-cache then you may lose that locality by the time the RPC callback runs. There is also some intrinsic cost associated with callback deferment that makes it significantly more expensive than a synchronous direct function call, even for lightweight RPC arguments.
So if the RPC arguments are large (e.g., a large upcxx::view) or otherwise expensive to serialize (e.g., lots of std::string or other containers requiring individual fine-grained allocations), and/or are called with very high frequency, then you might consider converting a loopback RPC into a direct function call, ideally passing the arguments by reference (assuming the callback can safely share the input argument data without copying it). Such a transformation would be prohibited by the library API semantics (i.e., because it's not transparent), but a user with knowledge of the application and callback semantics could do it safely in many circumstances and potentially get a big win under the right circumstances.
Several of the STL container types are automatically serialized, as are TriviallyCopyable types. For more complicated types there are several mechanisms for customizing serialization - see this guide section. Note that if you are transferring a container of data elements "consumed" by the callback, you should consider using view-based serialization to transfer those elements to reduce extraneous data copies (see spec for details).
I have defined some types that are not trivially copyable and thus need to be serialized. When I try to RMA an instance of my class, static_assert for TriviallySerializable concept fails.
The error you are getting refers to TriviallySerializable because the RMA operation you are attempting does not support the serialization you seek. Operations which are intended to be accelerated by network RMA hardware (such as rput, rget) can only move flat byte sequences, hence we assert the type given can be meaningfully moved that way. RPC's are not restricted in the types they transmit since the CPU is always involved, and we do indeed accept and serialize most std containers when given as RPC arguments. There are also now interfaces for defining custom serialization on your own types for use with RPC (see the spec for details).
How does the efficiency of passing a upcxx::view<T> argument to RPC compare to other mechanisms for passing the same TriviallySerializable T elements to an RPC callback?
Background info on view-based serialization.
A view argument over a TriviallySerializable type adds 8 bytes of serialized data payload to store the number of elements. Aside from that there are no additional overheads to using a view. There are no incremental costs associated with direct access to the network buffer from the RPC callback at the target.
If your alternative is to send those elements in a container like a std::vector, std::list, std::set, etc those also all add 8 bytes of element count in the serialized representation (std::array notably does not, as the element count is static info), but these containers all also add additional element data copies and allocation calls to construct the container at the target. So a view should never be worse than a container over the same elements, and can be significantly better.
If your alternative is to send each element as an individual unboxed RPC argument, that might be comparable to a view for a very small number of elements. But the view should quickly win once you amortize the extra 8 bytes on-the-wire and reap the efficiency benefit of passing the arguments to the callback function via pointer indirection instead of forcing the compiler to copy/move each element to the program stack (beyond the point where they all fit in registers).
The main benefits are:
- The data transfer is performed in a zero-copy manner on most HPC-relevant hardware (e.g. Cray networks or InfiniBand). Fewer payload copies means less CPU overheads tied up in communication, and these costs are especially relevant when the payload is large.
- On HPC-relevant hardware, the transfer is one-sided down to the hardware level. Specifically, the data transfer is offloaded to the network hardware on the remote end, meaning there are no communication delays associated with involvement of the remote CPU.
These same benefits also apply to upcxx::copy when applied to host memory or device memory with native memory kinds support.
Yes, with a caveat. UPC++ allows RMA transfers (rput and copy) to request target-side remote completion notification, sometimes known as "signaling put". This feature is described in this guide section.
Here's a simple example:
double *lp_src = ...;
global_ptr<double> gp_dst = ...;
upcxx::rput(lp_src, gp_dst, count,
remote_cx::as_rpc([](t1 f1, t2 f2){ ... }, a1, a2));The RMA data payload transfer (from lp_src to gp_dst) remains fully one-sided and zero-copy using RDMA hardware on networks that provide that capability (eg. Cray Aries and InfiniBand).
The RPC arguments (a1, a2 and the lambda itself) are subject to serialization semantics, so that part of the payload is not zero-copy; however in a well-written application this is typically small (under a few cache lines) so the extra copies are irrelevant to performance. Also as usual, the RPC callback will not execute until the target process enters UPC++ user-level progress.
If two or more RMA operations concurrently access the same memory location with no intervening synchronization and at least one of them is a write/put, that constitutes a data race, just like in shared-memory programming. An RMA data race is not immediately fatal, but any values returned and/or deposited in memory will be indeterminate, ie potentially garbage. Well-written programs should use synchronization to avoid data races.
UPC++ also provides remote atomic operations that guarantee atomicity in the presence of conflicting RMAs to locations in shared memory. These can be used in ways analogous to atomic operations in shared-memory programming to ensure deterministically correct behavior.
Is a UPC++ rank a thread or a process? How does UPC++ allow one rank tries to directly access the memory of another rank?
Like MPI, a UPC++ rank is a process. How they are created is platform dependent, but you can count on fork() being a popular case for non-supercomputers. Typically, UPC++ processes use process shared memory (POSIX shm_***) to communicate within a node. Several other OS-level shared memory bypass mechanisms are also supported - see GASNet documentation on "GASNet inter-Process SHared Memory (PSHM)" for details.
See docs/debugging for tips on debugging UPC++ codes.
Like MPI and other SPMD models, the maximum amount of potential parallelism has to be specified at job launch. Dynamically asking for more processes to join the job is not possible, but asking for more than you need up front and "waking" them dynamically is doable. Consider having all but rank=0 spinning in a loop on upcxx::progress(). When rank 0 wants to offload work to rank 1, it can send an rpc which could kick it out of its spin loop to do real work.
How do I launch a UPC++ program on multiple nodes, and particularly in the case when the nodes are connected by IB?
The UPC++ configure script should autodetect the availability of InfiniBand (done during a step called "configuring gasnet"). To build a program to run over infiniband, make sure that:
-
CXX=mpicxxis in your environment when calling the configure script. - Compile your program using
upcxx -network=ibv
Then to launch just use the upcxx-run script.
You have 3 options:
a. upcxx-run (which internally invokes gasnetrun_ibv) to perform ssh-based spawning.
- This option requires you to correctly setup password-less SSH authentication from at least your head node to all the compute nodes - this document describes how to do that in the context of BUPC (which also uses GASNet) and the information is analogous for UPC++
- It additionally requires that you pass the host names into the environment, e.g.
GASNET_SSH_SERVERS="host1 host2 host3... - The
gasnetrun_ibv -voption is often useful for troubleshooting site-specific problems that may arise here. - You can see more details of what upcxx-run is doing by setting
UPCXX_VERBOSE=1in the environment, and even more by manually passing -v to the underlyinggasnetrun_ibvcommand.
b. mpirun (possibly invoked from upcxx-run) - uses MPI for job spawn ONLY, then IBV for communication
- This requires UPC++/GASNet was configured, built and installed with MPI support (usually by setting
CXX=mpicxx) - Also requires that (non-GASNet) MPI programs spawn correctly via mpirun (and any MPI-implementation-specific tweaking required to make that work)
- It's also best to use TCP-based MPI if possible for this purpose, to prevent the MPI library from consuming IBV resources that won't be used by the app. There is more info on that topic in this document: https://gasnet.lbl.gov/dist/other/mpi-spawner/README
- mpirun often has the -v option to provide spawn status for troubleshooting
c. PMI spawning.
For more details about job spawning see this guide section
How to I bind processes to cores and ensure the binding is correct to maximize NUMA hardware resources?
The best means of core-binding is system-specific. Parallel job spawners such as SLURM srun and IBM jsrun have options to control core binding as part of parallel process launch, and on systems with those spawners that's usually the best place to start. This might mean a trial run with upcxx-run -show to get the necessary "starting point" for envvars and spawner command, before manually adding some core-binding options. Failing those, tools like hwloc's hwloc-bind can be invoked as a wrapper around your application to achieve specific binding, although it might require some scripting trickery to bind different ranks to distinct resources.
Either way on a system with the hwloc package installed, you can programmatically verify cores were bound as expected using code like the following, which prints the current core binding to stdout:
std::string s("echo ");
s += std::to_string(upcxx::rank_me()) + ": `hwloc-bind --get`";
system(s.c_str());For more details about job spawning see this guide section
I ran the following program on my cluster and the output arrived in an unexpected order. Does this mean barrier is broken?
printf("Hello world from rank %d\n", upcxx::rank_me());
fflush(stdout);
upcxx::barrier();
printf("Goodbye world from rank %d\n", upcxx::rank_me());
fflush(stdout);The issue here is that while upcxx::barrier() DOES synchronize the execution of all the processes, it does NOT necessarily flush all their stdout/stderr output all the way back to the central console on a cluster. The call to fflush(stdout) ensures that each process has flushed output from its private buffers to the file descriptor owned by each local kernel, however in a distributed-memory run this output stream still needs to travel over the network (eg via a socket connection) from the kernel of each node back to the login/batch node running the upcxx-run command before it reaches the console and is multiplexed with output from the other compute nodes to appear on your screen. upcxx::barrier() does not synchronize any part of that output-forwarding channel, so it's common to see such output "races" on distributed-memory systems. The same is true of MPI_Barrier() for the same reasons.
The only way to reliably workaround this issue is to perform all console output from a single process (conventionally rank 0). This is also a good idea for scalability reasons, as generally nobody wants to scroll through 100k copies of "hello world" ;-). It's also worth noting that console output usually vastly underperforms file system I/O with a good parallel file system, so the latter should always be preferred for large-volume I/O.
UPC++ provides built-in teams for the entire job (upcxx::world()) and for the processes co-located on the same physical node (upcxx::local_team()). There is no built-in singleton team for the calling process, but it's easy to construct one using team::create() -- see example below.
#include <upcxx/upcxx.hpp>
upcxx::team *team_self_p;
inline upcxx::team &self() { return *team_self_p; }
int main() {
upcxx::init();
int world_rank = upcxx::rank_me();
// setup a self-team:
team_self_p = new upcxx::team(upcxx::world().create(std::vector<int>{world_rank}));
// use the new team
UPCXX_ASSERT(self().rank_me() == 0);
UPCXX_ASSERT(self().rank_n() == 1);
upcxx::barrier(self());
upcxx::dist_object<int> dobj(world_rank, self());
UPCXX_ASSERT(dobj.fetch(0).wait() == world_rank);
// cleanup (optional)
self().destroy();
delete team_self_p;
upcxx::finalize();
}For more info on teams, see this guide section.