json -> scalar casts and array indexing are two culprits that showed up in some recent testing I've done, and together slowed a test down by about 2x.
The culprit seems to be this rule from https://wiki.postgresql.org/wiki/Inlining_of_SQL_functions:
- if the function is declared IMMUTABLE, then the expression must not invoke any non-immutable function or operator