Changing the font size with Readium CSS is limited because if the publication has absolute font sizes then enabling advanced options is necessary: basically disabling publisher's defaults.
A possible workaround would be to use native solutions when available since Readium CSS is platform-agnostic.
text-size-adjust is promising and seems to be available on both iOS and Android: https://caniuse.com/?search=text-size-adjust
Related issue: readium/css#34