x509asn1: cleanup and unify code layout

- rename 'n' to buflen in functions, and use size_t for them. Don't pass
  in negative buffer lengths.

- move most function comments to above the function starts like we use
  to

- remove several unnecessary typecasts (especially of NULL)

Reviewed-by: Patrick Monnerat
Closes #3582