Tag invalid objects instead of failing migrations

[rudolf]

Don't fail migrations if an invalid object is encountered, but tag object as invalid

1. Tag objects as “invalid” if they cause an exception when being migrated, but don’t fail the entire migration.
2. Log an error message informing administrators that there are invalid objects which require inspection. For each invalid object, provide an error stack trace to aid in debugging.
3. Administrators should be able to generate a migration report (similar to the one dry run migrations create) which is an NDJSON export of all objects tagged as “invalid”.
4. When an invalid object is read the Saved Objects client will throw an invalid object exception which should include a link to documentation to help administrators resolve migration bugs.
5. Educate Kibana developers to no longer simply write back an unmigrated document if an exception occurred. A migration should either successfully transform the object or throw.

Part of #52202

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[rudolf]

From Section 5.5 in the "Improve saved object migrations RFC" #66056

Although it seems preferable to not fail the entire migration because of a single saved object type's migration transform bug or a single invalid document this has several pitfalls:

• When an object fails to migrate the data for that saved object type becomes inconsistent. This could load to a critical feature being unavailable to a user leaving them with no choice but to downgrade.
• Because Kibana starts accepting traffic after encountering invalid objects a rollback will lead to data loss leaving users with no clean way to recover.

As a result we prefer to let an upgrade fail and making it easy for users to rollback until they can resolve the root cause.

[pgayvallet]

Reopening, as this may be required for some of the possible enhancements we could implement.