allow plugins to provide builtin goals and refactor BSP into backends

[tdyas]

Allow plugins to provide builtin goals via the newly-introduced builtin_goals function in plugin registration module.

Refactor the BSP (Build Server Protocol) support to use that feature by moving it out of the Pants core rules into several new backends.

[sureshjoshi]

What's the use case for a plug in providing "built-in goals" rather than regular goal rules?

Feels like BuiltinGoal would no-longer be the correct term for these.

[tdyas]

What's the use case for a plug in providing "built-in goals" rather than regular goal rules?

Feels like BuiltinGoal would no-longer be the correct term for these.

BuiltinGoal is probably misnamed since it is for goals which run without full option parsing (e.g., the help system). For BSPGoal, it needs to take over stdin and stdout when the Pants BSP implementation runs in "server" mode.

EarlyGoal is probably a better name.

I was not trying to rename BuiltinGoal with this PR though. Should I be?

[sureshjoshi]

I was not trying to rename BuiltinGoal with this PR though. Should I be?

Probably not.

But we should probably carve out a ticket/discussion about this - as once this lands, BuiltIn goal becomes part of the usable public API, need docs around naming, why to use this vs goal_rule, etc etc blah blah.

[tdyas]

Opened discussion: #20931