Converting post metadata to a better system

[sybrew]

Currently, post metadata is stored through individual keys.

This means there are a lot of entries stored in the database, and key collision is prominent.

Current issues:

1. Prone to key collisions.
2. There are loads of entries, increasing exponentially as we add more options.
3. Entries that don't start with an underscore, are shown as separated custom fields. Offenders:
   • exclude_from_archive
   • exclude_local_search
   • redirect
4. The data isn't grouped and therefore, requires more effort to find, debug, handle, etc.

To resolve this, we must upgrade the metadata from single entries to an array with a common (namespaced) key.

References:

• Now: Entity-attribute-value model
• Target: Key-value store

How we did the term meta upgrade:

In WordPress 4.4, new term metadata functionality was added. In TSF 2.7, the Term Metadata upgrade was handled and annotated.

This upgrade was done in one single request.

the-seo-framework/inc/functions/upgrade.php

Lines 117 to 131 in 2c24ef6

	/**
	* Upgrades term metadata for version 2701.
	*
	* @since 2.7.0
	*/
	function the_seo_framework_do_upgrade_2701() {
	$term_meta = get_option( 'autodescription-term-meta' );
	foreach ( (array) $term_meta as $term_id => $meta ) {
	add_term_meta( $term_id, THE_SEO_FRAMEWORK_TERM_OPTIONS, $meta, true );
	}
	update_option( 'the_seo_framework_upgraded_db_version', '2701' );
	}

This can't be done for posts.

Obstacles:

1. Term data was converted from a single database entry. Post data has to be converted from thousands to even millions of database entries.
2. TSF's post-meta API has to be prominently adopted, a must-use for developers so they won't see a change of functionality. This already exists: get_custom_field().
3. Old and new data both must be checked for; i.e. either through flags or through plain deletion if new data is inserted.

Propositions

1. Convert on post save, while maintaining two versions. This means:
   • All posts saves will convert data to the newer version, discarding the old.
2. Convert on-the-fly on post visit, while maintaining two versions. This means:
   • All posts visits will convert data to the newer version, discarding the old.
3. Upgrade loop. This means:
   • After plugin update, the user is prompted to upgrade posts.
   • When activated, it will loop through all N posts every single admin request until done.

Now, all propositions can work alongside. In all cases, new data will be tested for existence, and will then fall back to old.

Footnotes

When done follow-up: #48
Related functionality: #84

[sybrew]

Candidate for 3.1.0. But, I don't want to rush this as it's a vital component and it needs to be done perfectly.

[sybrew]

Maybe later, it's not pressing.

[sybrew]

In TSF 4.0, we revised how post-metadata is handled, with this issue in mind.
See the post-data.class.php file.

[sybrew]

When we move to this method of storing data, plugins like "Better Search and Replace", and possibly "WP-Optimize", will annihilate the metadata as they pass over it.

Here's one recent finding, where "Better Search and Replace" replaces Extension Manager's neatly-packed metadata with (string) "1".

[sybrew]

The next major release provides a basis that allows migrating data. Some considerations:

1. The migration must be user-invoked if the site has over 200 posts. Otherwise, this can bring unexpected downtime.
2. When the migration is activated manually, we should point the admin to an interface. This interface should show, quite verbosely, the number of posts marked for migrating and the process -- either via event-stream or a browser-invoked AJAX loop (which is probably easiest for this one-time matter).
3. When the migration is pending, a flag must be set to fetch data only from the old system.
4. When the migration is active, a flag must be set to fetch data from both old and new systems.
5. When the migration is completed, a flag must be set to fetch data only to the new system.
6. We should batch 50 to 100 posts for every request. Data cannot be crossed, so multithreading shouldn't cause an issue.
7. When the migration is active, we should notify how many posts have been migrated and how many are left. This is to alleviate concerns surrounding slowdowns.
8. We should emit a notification once the migration is completed.
9. We won't maintain the "redirect" index. Too few sites rely on this for other reasons than TSF. However, we should provide a snippet to revert the process.

[sybrew]

When this is resolved, we can implement filter sanitize_post_meta_{$meta_key}, allowing the sanitization of our metadata, even when others try to augment it.

For now, \The_SEO_Framework\Data\Filter\Plugin::filter_meta_update() is unhooked (commit pending), and the data still won't be sanitized on-save outside of our plugin saving API.

[sybrew]

When we migrate, note that WPML may have stored translation data, which we should also migrate.
WPML does not support a key-value store for post and term meta, but it does for site options. We need to reach out to them about this.

Ref: https://wpml.org/documentation/getting-started-guide/translating-custom-fields/.

[sybrew]

To #185 (comment):

From WPML's staff:

Did you check this documentation page about XML config for custom fields?
You should manage to define which sub-keys to translate inside a custom field array value.

Example:

<wpml-config>
  <custom-fields>
    <custom-field action="translate">with_attributes</custom-field>
    <custom-field action="translate">with_deep_attributes</custom-field>
    <custom-field action="translate">no_attributes</custom-field>
  </custom-fields>
  <custom-fields-texts>
    <key name="with_attributes">
        <key name="attribute1" />
        <key name="attribute2" />
    </key>
    <key name="with_deep_attributes">
        <key name="attribute1" />
        <key name="attribute2">
          <key name="level1">
          <key name="level2">
          </key>
          </key>
        </key>
    </key>
  </custom-fields-texts>
</wpml-config>