anything that is a programming error (violating "contract" pre/post/invariant conditions), such as calling a function with invalid arguments or corrupted data-structures, should be checked with assert statements. These should never happen for valid code, so we don't want to waste cpu cycles on them in a production release, but we want to catch them in debug builds.
Anything that is very rare and basically unrecoverable for the whole program, like malloc failures, should be rs_fatal.
Anything that is an error caused by external things beyond the program's control, like bad input data, or that break librsync in ways that shouldn't kill the whole program, should be caught and return an error code for the program to handle as it sees fit.
Note I fixed quite a few of these in pull #88. A quick search shows not many left that are not meeting the criteria above. The only ones left that need changing are things that are checking programming errors and thus could be replaced with asserts for efficiency. This means we don't call rs_fatal() in any places that would be considered "not well-behaved library behaviour" any more.