A basic forum specific search is implemented using the SilverStripe ORM layer.
The search form is part of the ForumHeader.ss include, override this in your theme of you need to customise this.
The forum search results can be styled by creating
ForumHolder_search.ss layout for your theme.
For a more advanced search implementation you may wish to look into the fulltextsearch module.