I'm pretty sure this happens because the test for the ssl library still works because it finds a system one installed and uses that (in one of the default paths). So while a check for the dir existing would catch your specific case if you truly point to a missing directory, it would still work like this if you point to an empty directory...
I can't think of any easy ways to detect this. The linker has a default path that might make it find another library if the given path isn't working. To detect that, we have to add some custom logic that verifies the given path to make sure it actually contains a (working) openssl installation and that does feel very error-prone.
Or do you have any ideas on how this process could be improved?
Well, do we need to know it is a working installation in the specified directory? Wouldn't it be enough that we find at least the "entry header file" in the directory? If the user points to a broken installation, that'll likely become obvious at a later stage when it doesn't compile or otherwise breaks.
Other than that, I can only think of of documenting the possible pitfall.
When given a prefix, the $PREFIX_OPENSSL/lib/openssl.pc or
$PREFIX_OPENSSL/include/openssl/ssl.h files must be present or cause an
error. Helps users detect when giving configure the wrong path.
Reported-by: Oleg Pudeyev
Assisted-by: Per Malmberg