-
Notifications
You must be signed in to change notification settings - Fork 1
FAQ
Q. How should I handle serialization of user-defined classes?
A. Several of the STL container types are automatically serialized, as are TriviallyCopyable types. For more complicated types there are several mechanisms for customizing serialization - see this guide section. Note that if you are transferring a container of data elements "consumed" by the callback, you should consider using view-based serialization to transfer those elements to reduce extraneous data copies (see spec for details).
Q. I have defined some types that are not trivially copyable (they contain std containers) and thus they need to be serialized. upcxx specification says that I should be able to do the same, but when I try to rget an instance of my class, static_assert for TriviallySerializable concept fails.
A. The error you are getting refers to TriviallySerializable, the RMA operation you are attempting does not support the serialization you seek. Operations which are intended to be accelerated by network RMA hardware (such as rput, rget) can only move flat byte sequences, hence we assert the type given can be meaningfully moved that way. RPC's are not restricted in the types they transmit since the CPU is always involved, and we do indeed accept and serialize most std containers when given as RPC arguments. There are also now interfaces for defining custom serialization on your own types for use with RPC (see the spec for details).
Q. Is a rank a thread or a process? Are they created using Pthreads or Linux system call fork()? If each rank is a process, and I run UPC++ in some machine which has security enforced (accessing another process's memory is not allowed), does UPC++ crash when one rank tries to access the memory of another rank?
A. Like MPI, a rank is a process. How they are created is platform dependent, but you can count on fork() being a popular case for non-supercomputers. Typically, UPC++ processes use process shared memory (POSIX shm_***) to communicate within a node. Several other OS-level shared memory bypass mechanisms are also supported - see GASNet documentation on "GASNet inter-Process SHared Memory (PSHM)" for details.
Q. How do I launch a UPC++ program on multiple nodes, and particularly in the case when the nodes are connected by IB?
A. The UPC++ configure script should autodetect the availability of InfiniBand (done during a step called "configuring gasnet"). To build a program to run over infiniband, make sure that:
-
CXX=mpicxxis in your environment when calling the configure script. - Compile your program using
upcxx -network=ibv
Then to launch just use the upcxx-run script.
Q. When I conjoin futures using when_all(), it appears that progress is made only when wait() is called and that the operations complete in an arbitrary order. In what order do the operations complete? Why do the conjoined operations complete in less time than an explicitly serial ordering? Is there a possibility of a race condition? Does each future-bearing operation execute in its own thread.
A. Futures progress sequentially on the calling thread and in unspecified order. UPC++ futures are not a tool for expressing parallelism, but instead non-deterministic sequential behavior. Futures allow a thread to react to events in the order they actually occur, since the order in which communication operations complete is not deterministic. While UPC++ does support multiple threads within each process, and each thread can be managing its own futures, the same future must never be accessed concurrently by multiple threads. Also, UPC++ will never "spawn" new threads. See the section "Futures and Promises" of the UPC++ specification.
Q. Does a barrier synchronize RMA operations (rput/rget)?
A. No. UPC++ provides per-operation completions to this end. See this guide section
Q. Are UPC++ futures thread-safe? A. No.
Q. What is the strategy for using a debugger with UPC++?
A. See docs/debugging for tips on debugging UPC++ codes.
Q. Can a UPC++ program dyanmically add more processes?
A. Like MPI and other SPMD models, the maximum amount of potential parallelism has to be specified at job launch. Dynamically asking for more processes to join the job is not possible, but asking for more than you need up front and "waking" them dynamically is doable. Consider having all but rank=0 spinning in a loop on upcxx::progress(). When rank 0 wants to offload work to rank 1, it can send an rpc which could kick it out of its spin loop to do real work.
Q. How do I launch distributed UPC++ jobs with Infiniband?
A. You have 3 options:
a. upcxx-run (which internally invokes gasnetrun_ibv) to perform ssh-based spawning.
- This option requires you to correctly setup password-less SSH authentication from at least your head node to all the compute nodes - this document describes how to do that in the context of BUPC (which also uses GASNet) and the information is analogous for UPC++
- It additionally requires that you pass the host names into the environment, e.g.
GASNET_SSH_SERVERS="host1 host2 host3... - The
gasnetrun_ibv -voption is often useful for troubleshooting site-specific problems that may arise here. - You can see more details of what upcxx-run is doing by setting
UPCXX_VERBOSE=1in the environment, and even more by manually passing -v to the underlyinggasnetrun_ibvcommand.
b. mpirun (possibly invoked from upcxx-run) - uses MPI for job spawn ONLY, then IBV for communication
- This requires UPC++/GASNet was configured, built and installed with MPI support (usually by setting
CXX=mpicxx) - Also requires that (non-GASNet) MPI programs spawn correctly via mpirun (and any MPI-implementation-specific tweaking required to make that work)
- It's also best to use TCP-based MPI if possible for this purpose, to prevent the MPI library from consuming IBV resources that won't be used by the app. There is more info on that topic in this document: https://gasnet.lbl.gov/dist/other/mpi-spawner/README
- mpirun often has the -v option to provide spawn status for troubleshooting
c. PMI spawning.
For more details see this guide section
Q. Will calling chrono sleeping flip a future to the ready state?
A. No. It is a requirement that upcxx::progress() be called for futures to flip to their ready state. No amount of chrono sleeping will flip a future even if the transfer has actually completed.
Q. How to I bind processes to cores and ensure the binding is correct to maximize NUMA hardware resources?
A. The best means of core-binding is system-specific. Parallel job spawners such as SLURM srun and IBM jsrun have options to control core binding as part of parallel process launch, and on systems with those spawners that's usually the best place to start. This might mean a trial run with upcxx-run -show to get the necessary "starting point" for envvars and spawner command, before manually adding some core-binding options. Failing those, tools like hwloc's hwloc-bind can be invoked as a wrapper around your application to achieve specific binding, although it might require some scripting trickery to bind different ranks to distinct resources.
Either way on a system with the hwloc package installed, you can programmatically verify cores were bound as expected using code like the following, which prints the current core binding to stdout:
std::string s("echo ");
s += std::to_string(upcxx::rank_me()) + ": `hwloc-bind --get`";
system(s.c_str());Q. Does upcxx do any optimization if I happen to call an rpc destined to rank_me()? Do I need to explicitly do anything to make that more efficient?
A. There is an optimization applied to all targets in the local_team() (shared-memory bypass avoids one payload copy for large payloads using the rendezvous-get protocol), but nothing specific to same-process "loopback" (rank_me).
If you think same-process/loopback RPCs comprise a significant fraction of your RPC traffic, you should probably consider that serialization semantics require copying the argument payload at least once, even for a loopback RPC. Also, the progress semantics require deferring callback invocation until the next user-level progress, so if the input arguments are hot in-cache then you may lose that locality by the time the RPC callback runs. There is also some intrinsic cost associated with callback deferment that makes it significantly more expensive than a synchronous direct function call, even for lightweight RPC arguments.
So if the RPC arguments are large (e.g., a large upcxx::view) or otherwise expensive to serialize (e.g., lots of std::string or other containers requiring individual fine-grained allocations), and/or are called with very high frequency, then you might consider converting a loopback RPC into a direct function call, ideally passing the arguments by reference (assuming the callback can safely share the input argument data without copying it). Such a transformation would be prohibited by the library API semantics (i.e., because it's not transparent), but a user with knowledge of the application and callback semantics could do it safely in many circumstances and potentially get a big win under the right circumstances.
Q. I ran the following program on my cluster and the output arrived in an unexpected order. Does this mean barrier is broken?
printf("Hello world from rank %d\n", upcxx::rank_me());
fflush(stdout);
upcxx::barrier();
printf("Goodbye world from rank %d\n", upcxx::rank_me());
fflush(stdout);A. The issue here is that while upcxx::barrier() DOES synchronize the execution of all the processes, it does NOT necessarily flush all their stdout/stderr output all the way back to the central console on a cluster. The call to fflush(stdout) ensures that each process has flushed output from its private buffers to the file descriptor owned by each local kernel, however in a distributed-memory run this output stream still needs to travel over the network (eg via a socket connection) from the kernel of each node back to the login/batch node running the upcxx-run command before it reaches the console and is multiplexed with output from the other compute nodes to appear on your screen. upcxx::barrier() does not synchronize any part of that output-forwarding channel, so it's common to see such output "races" on distributed-memory systems. The same is true of MPI_Barrier() for the same reasons.
The only way to reliably workaround this issue is to perform all console output from a single process (conventionally rank 0). This is also a good idea for scalability reasons, as generally nobody wants to scroll through 100k copies of "hello world" ;-). It's also worth noting that console output usually vastly underperforms file system I/O with a good parallel file system, so the latter should always be preferred for large-volume I/O.
Q. Can you make a RPC call from within a RPC callback? When we try to wait on the nested RPC we get an error/hang.
A. You can absolutely launch RPC or RMA from within an RPC callback. However you cannot wait on the completion of communication from within the RPC callback. If you run in debug mode you'll get an error message explaining that, e.g.: "You have attempted to wait() on a non-ready future within upcxx progress, this is prohibited because it will never complete."
So if you launch communication from RPC callback context you need to use UPC++'s other completion and asynchrony features to synchronize completion instead of calling future::wait(). There are many different ways to do this, and the best choice depends on the details of what the code is doing. For example, one could request future-based completion on that op and either stash the resulting future in storage outside the callback's stack frame, or better yet schedule a completion callback on the future using future::then() with a continuation of whatever you would have done after wait. There are also operations (like rpc_ff) and completion types (remote_cx::as_rpc) that signal completion at the remote side, so you perform synchronization at the target instead of the initiator.
Q. How does the efficiency of passing a upcxx::view<T> argument to RPC compare to other mechanisms for passing the same TriviallySerializable T elements to an RPC callback?
A. A view argument over a TriviallySerializable type adds 8 bytes of serialized data payload to store the number of elements. Aside from that there are no additional overheads to using a view. There are no incremental costs associated with direct access to the network buffer from the RPC callback at the target.
If your alternative is to send those elements in a container like a std::vector, std::list, std::set, etc those also all add 8 bytes of element count in the serialized representation (std::array notably does not, as the element count is static info), but these containers all also add additional element data copies and allocation calls to construct the container at the target. So a view should never be worse than a container over the same elements, and can be significantly better.
If your alternative is to send each element as an individual unboxed RPC argument, that might be comparable to a view for a very small number of elements. But the view should quickly win once you amortize the extra 8 bytes on-the-wire and reap the efficiency benefit of passing the arguments to the callback function via pointer indirection instead of forcing the compiler to copy/move each element to the program stack (beyond the point where they all fit in registers).