The API changes between each major GHC release. The changes may not be extensive, but they are often not documented in the release notes
The documentation is not always very detailed
Some flags can only be set before starting a GHC session, and a GHC session cannot be restarted. This explains the design of buildwrapper: it's an executable that you call for each operation, instead of a long running process. So each call using the GHC API starts a new session, perform the operation, and returns. Not great for performance, but see next point.
GHC API uses a lot of memory as you reload files. Things like instances and template haskell make things worse. When EclipseFP used scion, which was a long running process, working on a Yesod app for 2 hours caused scion to use 1Gb of memory.
Navigating the AST can be dangerous: depending on the phases you've run on this AST, not all objects are filled in, and accessing an unfilled object result in a panic (see the comments inside the source code of ghc-syb-utils).