Merge queue building too many things #10132
Description
Right now seems like we are building things that are not involved in the commit in question.
I thought this was due to comparing to main and so if you have anything in the queue ahead you also build what that woudl trigger but noticed this morning merging a pr it did build everything without anything ahead.
For typespec workspace we also could migrate to github actions which would simplify all of this.