Use SmallVec for some Vecs used during frame building that are usual… …
…ly small and don't need an allocation