Every function (default, image optimization, middleware ... ) should have an install option to install native dependencies for the correct arch. This would also allow to install sharp on the default/splitted functions if needed.
Something like that :
install : {
packages: ['sharp', ...],
arch: 'arm'|'x64',
}