Fix github issue #899
which came from e-support UOY-859712.

The problem was that the vlen_max parameter
to libsrc/var.c#NC_check_vlen was of type size_t.
However, it is being called, sometimes, with values
of size X_INT64_MAX. The resulting truncation was
causing dimension failures as noted in the e-support
report.

Fix is to change the vlen_max argument (and some
local variables in NC_check_vlen) to be of declared
as unsigned long long.