Add batch facility to perform reconciliation with a slave server

[bobbingwide]

With the current logic I'm loathe to change content on a slave since this is not reflected in the master. I want to be able to reconcile changes applied to a slave to the original master.

[bobbingwide]

To assist in the development and testing I need to improve the error reporting capability in the libs\class-oik-remote.php shared library.

[bobbingwide]

There's a problem with implementing the reconciliation with the current data. When a post is cloned the cloned date is set to the post_modified_gmt of the post being cloned.
When the post is being updated in the server then the timestamp of the updated slave post can become different from the timestamp of the post being cloned.
So the cloned date and post_modified_gmt values are not the same from the word go.
This makes it a lot trickier to tell if the post has actually been changed since it was cloned!

A pragmatic workaround may be to allow for up to a minute's difference between timestamps.

[bobbingwide]

The first pass routine uses an updated version of the request_mapping logic ( from class-oik-clone-reset-ids.php ) to obtain the mapping for the slave, returning the cloned timestamp and the post_modified_gmt of the slave post. This is then matched with the original post and the post_modified_gmt timestamps compared.

The rather verbose output reports what might need to be done.

[bobbingwide]

The latest version performs a basic pull from a remote server ( not a WPMS site ) with some limitations

• It doesn't pull attached files
• It doesn't update the cloned timestamp on the slave server.

[bobbingwide]

Since this is a major change the new version will need to be 2.0.0.
The new code should be able to cater for the fact that the remote server is running a lower level version, which may not support the AJAX request "oik-clone-pull".

[bobbingwide]

Re: #34 (comment)

It doesn't update the cloned timestamp on the slave server.

The logic has now been changed to reset the cloned timestamp on the master after performing the pull, setting it to the original cloned date.

[bobbingwide]

The logic that requests the mappings on the server should take into account the fact that a post may have been cloned to multiple targets, which may have happened inadvertently.

It would appear that the inadvertent cloning to a new post is also producing a Warning:

[12-May-2019 11:04:04 UTC] PHP Warning:  addslashes() expects parameter 1 to be string, object given in C:\apache\htdocs\wp52\wp-includes\formatting.php on line 5312
[12-May-2019 11:04:04 UTC] PHP Stack trace:
[12-May-2019 11:04:04 UTC] PHP   1. {main}() C:\apache\htdocs\wp52\wp-admin\admin-ajax.php:0
[12-May-2019 11:04:04 UTC] PHP   2. do_action() C:\apache\htdocs\wp52\wp-admin\admin-ajax.php:187
[12-May-2019 11:04:04 UTC] PHP   3. WP_Hook->do_action() C:\apache\htdocs\wp52\wp-includes\plugin.php:465
[12-May-2019 11:04:04 UTC] PHP   4. WP_Hook->apply_filters() C:\apache\htdocs\wp52\wp-includes\class-wp-hook.php:310
[12-May-2019 11:04:04 UTC] PHP   5. oik_clone_nopriv_oik_clone_post() C:\apache\htdocs\wp52\wp-includes\class-wp-hook.php:286
[12-May-2019 11:04:04 UTC] PHP   6. oik_clone_lazy_clone_post() C:\apache\htdocs\wordpress\wp-content\plugins\oik-clone\oik-clone.php:258
[12-May-2019 11:04:04 UTC] PHP   7. oik_clone_attempt_import() C:\apache\htdocs\wordpress\wp-content\plugins\oik-clone\admin\oik-clone-clone.php:47
[12-May-2019 11:04:04 UTC] PHP   8. oik_clone_insert_post() C:\apache\htdocs\wordpress\wp-content\plugins\oik-clone\admin\oik-clone-clone.php:210
[12-May-2019 11:04:04 UTC] PHP   9. wp_slash() C:\apache\htdocs\wordpress\wp-content\plugins\oik-clone\admin\oik-clone-actions.php:327
[12-May-2019 11:04:04 UTC] PHP  10. addslashes() C:\apache\htdocs\wp52\wp-includes\formatting.php:5312

[bobbingwide]

#34 (comment)

There's a problem with implementing the reconciliation with the current data. When a post is cloned the cloned date is set to the post_modified_gmt of the post being cloned.
When the post is being updated in the server then the timestamp of the updated slave post can become different from the timestamp of the post being cloned.
So the cloned date and post_modified_gmt values are not the same from the word go.
This makes it a lot trickier to tell if the post has actually been changed since it was cloned!

A pragmatic workaround may be to allow for up to a minute's difference between timestamps.

A better fix to this is to implement the wp_insert_post_data filter and reset post_modified_gmt and post_modified date to the values passed from the original $postarr.

There will still be problems after a pull. The cloned dates need to be adjusted on both the master and the server.

[bobbingwide]

For attachments we also need to filter wp_insert_attachment_data.

[bobbingwide]

For pulling attachments, we need to consider what happens to the attached files. This should only become a problem when an image (or other file type ) has been uploaded directly to the server to create new content - i.e. not cloned.

... and the code does not yet cater for this situation.

[bobbingwide]

There will still be problems after a pull. The cloned dates need to be adjusted on both the master and the server.

To resolve this problem we add a subsequent AJAX request that updates the cloned information on the slave server to reflect fact that the master post, that has been pulled into the target, now has a new cloned timestamp equivalent to the post_modified_gmt.

The AJAX request is oik_clone_update_slave_target passing the following fields:

Field	Which is
slave_id	Post ID of the slave post that's been pulled.
master	URL of the master site
master_id	Post ID of the target post in the master
modified	post_modified_gmt of the newly imported post

With the changes for the wp_insert_post_data and wp_insert_attachment_data filters the post_modified_gmt and cloned timestamps should now match.

It may be possible to use this new AJAX request to resolve the mismatches.

[bobbingwide]

When there are a lot of posts the echo() method is far too verbose. It would be nice to get a summary of the actions that will be performed when using dry-run=y with verbose=n.

Then, once we know what sort of changes will be made we can run the reconciliation with dry-run=n and verbose=y.

Also add logic to determine which post types to clone, enabling the order to be chosen.

Here are a couple of basic .bat routines to start with:
reconciled.bat - dry run

cd \apache\htdocs\wordpress\wp-content\plugins\oik-clone\admin
php C:\apache\htdocs\wordpress\wp-content\plugins\oik-batch\oik-wp.php class-oik-clone-reconcile.php https://s.b/wp52 dry-run=y post-type=post,page verbose=n

reconcile.bat - the real mccoy

cd \apache\htdocs\wordpress\wp-content\plugins\oik-clone\admin
php C:\apache\htdocs\wordpress\wp-content\plugins\oik-batch\oik-wp.php class-oik-clone-reconcile.php https://s.b/wp52 dry-run=n post-type=post,page verbose=y

These routines will need to be parameterised / copied & cobbled for cloning from
c:\apache\htdocs\wp-a2z ( locally blocks.wp.a2z ) to https://blocks.wp-a2z.org

[bobbingwide]

In order to properly perform reconciliation we need to be able to list all the cloneable content on the slave, and pull any content that's missing to the master.

This query to list the posts needs to take into account attachments, which may have a status of inherit.

Note: This logic may end up pulling posts that were deleted from the master but not from the slave.
The pull logic needs to be able to download the attachments

[bobbingwide]

Having applied a couple of fixes, I've now alpha tested v2.0.0-alpha-20190602 with my local system blocks.wp.a2z as the master and https://blocks.wp-a2z.org as the slave.
I reconciled the attachments, oik-plugins and block CPTs.

Post type	local count	slave count	Comments
attachment	199	93	Hmm! See below
oik-plugins	30	29	Gutenberg-renamed has not yet been cloned
block	264	262	GT3 Photo & Video Gallery and another couple of accidentally created blocks are not yet cloned.

Importing of attachments seemed to duplicate content.
Some of the attachments listed on the slave were showing as attachments linked to draft posts on the local master.
There are now 189 attachments in the local master. weconciled lists all 93 on the slave as being reconciled. The other 96 attachments have not been cloned.

[bobbingwide]

Re:

Note: This logic may end up pulling posts that were deleted from the master but not from the slave.

and

Some of the attachments listed on the slave were showing as attachments linked to draft posts on the local master.

Because the attachments were attached to draft posts, then the logic in get_posts() reported that it did not find the attachment. Consequently another attachment was created. We should try to find a better way to deal with this scenario.

Similarly, I imagine this applies to the parent post. What should happen when a local post is not published?

[bobbingwide]

Once all the slave posts have been dealt with we should then turn to all the master posts that have been published but not yet cloned. The question is... do I really want to have the system cloning every published post to the selected slave during reconciliation?

In the case of a set of new block CPTs having been created for a plugin then it would be nice to have the whole lot cloned in one fell swoop, especially if there are a lot of them.
But there may be other content that was created locally that we don’t want to clone.
Do we need to be able to identify in advance whether or not content should be cloned to a particular server? If so, this should be addressed in a separate issue.

[bobbingwide]

When there are a lot of posts the echo() method is far too verbose.

Even when verbose=n is set we still get lots of lines starting "None" which makes it harder to see which posts are going to be pushed and which are going to be pulled.
Maybe we should add a terse=y option to further reduce the output.

[bobbingwide]

oik-clone v2.0.0 has been released. Closing this issue even though there are no PHPUnit tests.

"Life's too short to not be flippant sometimes"