Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add batch facility to perform reconciliation with a slave server #34

Open
bobbingwide opened this issue Apr 21, 2019 · 17 comments

Comments

1 participant
@bobbingwide
Copy link
Owner

commented Apr 21, 2019

With the current logic I'm loathe to change content on a slave since this is not reflected in the master. I want to be able to reconcile changes applied to a slave to the original master.

@bobbingwide bobbingwide self-assigned this Apr 21, 2019

@bobbingwide

This comment has been minimized.

Copy link
Owner Author

commented Apr 23, 2019

To assist in the development and testing I need to improve the error reporting capability in the libs\class-oik-remote.php shared library.

@bobbingwide

This comment has been minimized.

Copy link
Owner Author

commented Apr 23, 2019

There's a problem with implementing the reconciliation with the current data. When a post is cloned the cloned date is set to the post_modified_gmt of the post being cloned.
When the post is being updated in the server then the timestamp of the updated slave post can become different from the timestamp of the post being cloned.
So the cloned date and post_modified_gmt values are not the same from the word go.
This makes it a lot trickier to tell if the post has actually been changed since it was cloned!

A pragmatic workaround may be to allow for up to a minute's difference between timestamps.

@bobbingwide

This comment has been minimized.

Copy link
Owner Author

commented Apr 23, 2019

The first pass routine uses an updated version of the request_mapping logic ( from class-oik-clone-reset-ids.php ) to obtain the mapping for the slave, returning the cloned timestamp and the post_modified_gmt of the slave post. This is then matched with the original post and the post_modified_gmt timestamps compared.

The rather verbose output reports what might need to be done.

@bobbingwide

This comment has been minimized.

Copy link
Owner Author

commented Apr 23, 2019

The latest version performs a basic pull from a remote server ( not a WPMS site ) with some limitations

  • It doesn't pull attached files
  • It doesn't update the cloned timestamp on the slave server.
@bobbingwide

This comment has been minimized.

Copy link
Owner Author

commented Apr 23, 2019

Since this is a major change the new version will need to be 2.0.0.
The new code should be able to cater for the fact that the remote server is running a lower level version, which may not support the AJAX request "oik-clone-pull".

@bobbingwide

This comment has been minimized.

Copy link
Owner Author

commented May 12, 2019

Re: #34 (comment)

It doesn't update the cloned timestamp on the slave server.

The logic has now been changed to reset the cloned timestamp on the master after performing the pull, setting it to the original cloned date.

@bobbingwide

This comment has been minimized.

Copy link
Owner Author

commented May 12, 2019

The logic that requests the mappings on the server should take into account the fact that a post may have been cloned to multiple targets, which may have happened inadvertently.

It would appear that the inadvertent cloning to a new post is also producing a Warning:

[12-May-2019 11:04:04 UTC] PHP Warning:  addslashes() expects parameter 1 to be string, object given in C:\apache\htdocs\wp52\wp-includes\formatting.php on line 5312
[12-May-2019 11:04:04 UTC] PHP Stack trace:
[12-May-2019 11:04:04 UTC] PHP   1. {main}() C:\apache\htdocs\wp52\wp-admin\admin-ajax.php:0
[12-May-2019 11:04:04 UTC] PHP   2. do_action() C:\apache\htdocs\wp52\wp-admin\admin-ajax.php:187
[12-May-2019 11:04:04 UTC] PHP   3. WP_Hook->do_action() C:\apache\htdocs\wp52\wp-includes\plugin.php:465
[12-May-2019 11:04:04 UTC] PHP   4. WP_Hook->apply_filters() C:\apache\htdocs\wp52\wp-includes\class-wp-hook.php:310
[12-May-2019 11:04:04 UTC] PHP   5. oik_clone_nopriv_oik_clone_post() C:\apache\htdocs\wp52\wp-includes\class-wp-hook.php:286
[12-May-2019 11:04:04 UTC] PHP   6. oik_clone_lazy_clone_post() C:\apache\htdocs\wordpress\wp-content\plugins\oik-clone\oik-clone.php:258
[12-May-2019 11:04:04 UTC] PHP   7. oik_clone_attempt_import() C:\apache\htdocs\wordpress\wp-content\plugins\oik-clone\admin\oik-clone-clone.php:47
[12-May-2019 11:04:04 UTC] PHP   8. oik_clone_insert_post() C:\apache\htdocs\wordpress\wp-content\plugins\oik-clone\admin\oik-clone-clone.php:210
[12-May-2019 11:04:04 UTC] PHP   9. wp_slash() C:\apache\htdocs\wordpress\wp-content\plugins\oik-clone\admin\oik-clone-actions.php:327
[12-May-2019 11:04:04 UTC] PHP  10. addslashes() C:\apache\htdocs\wp52\wp-includes\formatting.php:5312
@bobbingwide

This comment has been minimized.

Copy link
Owner Author

commented May 14, 2019

#34 (comment)

There's a problem with implementing the reconciliation with the current data. When a post is cloned the cloned date is set to the post_modified_gmt of the post being cloned.
When the post is being updated in the server then the timestamp of the updated slave post can become different from the timestamp of the post being cloned.
So the cloned date and post_modified_gmt values are not the same from the word go.
This makes it a lot trickier to tell if the post has actually been changed since it was cloned!

A pragmatic workaround may be to allow for up to a minute's difference between timestamps.

A better fix to this is to implement the wp_insert_post_data filter and reset post_modified_gmt and post_modified date to the values passed from the original $postarr.

There will still be problems after a pull. The cloned dates need to be adjusted on both the master and the server.

@bobbingwide

This comment has been minimized.

Copy link
Owner Author

commented May 14, 2019

For attachments we also need to filter wp_insert_attachment_data.

@bobbingwide

This comment has been minimized.

Copy link
Owner Author

commented May 14, 2019

For pulling attachments, we need to consider what happens to the attached files. This should only become a problem when an image (or other file type ) has been uploaded directly to the server to create new content - i.e. not cloned.

... and the code does not yet cater for this situation.

@bobbingwide

This comment has been minimized.

Copy link
Owner Author

commented May 15, 2019

There will still be problems after a pull. The cloned dates need to be adjusted on both the master and the server.

To resolve this problem we add a subsequent AJAX request that updates the cloned information on the slave server to reflect fact that the master post, that has been pulled into the target, now has a new cloned timestamp equivalent to the post_modified_gmt.

The AJAX request is oik_clone_update_slave_target passing the following fields:

Field Which is
slave_id Post ID of the slave post that's been pulled.
master URL of the master site
master_id Post ID of the target post in the master
modified post_modified_gmt of the newly imported post

With the changes for the wp_insert_post_data and wp_insert_attachment_data filters the post_modified_gmt and cloned timestamps should now match.

It may be possible to use this new AJAX request to resolve the mismatches.

@bobbingwide

This comment has been minimized.

Copy link
Owner Author

commented May 26, 2019

When there are a lot of posts the echo() method is far too verbose. It would be nice to get a summary of the actions that will be performed when using dry-run=y with verbose=n.

Then, once we know what sort of changes will be made we can run the reconciliation with dry-run=n and verbose=y.

Also add logic to determine which post types to clone, enabling the order to be chosen.

Here are a couple of basic .bat routines to start with:
reconciled.bat - dry run

cd \apache\htdocs\wordpress\wp-content\plugins\oik-clone\admin
php C:\apache\htdocs\wordpress\wp-content\plugins\oik-batch\oik-wp.php class-oik-clone-reconcile.php https://s.b/wp52 dry-run=y post-type=post,page verbose=n

reconcile.bat - the real mccoy

cd \apache\htdocs\wordpress\wp-content\plugins\oik-clone\admin
php C:\apache\htdocs\wordpress\wp-content\plugins\oik-batch\oik-wp.php class-oik-clone-reconcile.php https://s.b/wp52 dry-run=n post-type=post,page verbose=y

These routines will need to be parameterised / copied & cobbled for cloning from
c:\apache\htdocs\wp-a2z ( locally blocks.wp.a2z ) to https://blocks.wp-a2z.org

@bobbingwide

This comment has been minimized.

Copy link
Owner Author

commented May 31, 2019

In order to properly perform reconciliation we need to be able to list all the cloneable content on the slave, and pull any content that's missing to the master.

This query to list the posts needs to take into account attachments, which may have a status of inherit.

Note: This logic may end up pulling posts that were deleted from the master but not from the slave.
The pull logic needs to be able to download the attachments

bobbingwide added a commit that referenced this issue Jun 2, 2019

bobbingwide added a commit that referenced this issue Jun 2, 2019

bobbingwide added a commit that referenced this issue Jun 2, 2019

@bobbingwide

This comment has been minimized.

Copy link
Owner Author

commented Jun 3, 2019

Having applied a couple of fixes, I've now alpha tested v2.0.0-alpha-20190602 with my local system blocks.wp.a2z as the master and https://blocks.wp-a2z.org as the slave.
I reconciled the attachments, oik-plugins and block CPTs.

Post type local count slave count Comments
attachment 199 93 Hmm! See below
oik-plugins 30 29 Gutenberg-renamed has not yet been cloned
block 264 262 GT3 Photo & Video Gallery and another couple of accidentally created blocks are not yet cloned.

Importing of attachments seemed to duplicate content.
Some of the attachments listed on the slave were showing as attachments linked to draft posts on the local master.
There are now 189 attachments in the local master. weconciled lists all 93 on the slave as being reconciled. The other 96 attachments have not been cloned.

@bobbingwide

This comment has been minimized.

Copy link
Owner Author

commented Jun 6, 2019

Re:

Note: This logic may end up pulling posts that were deleted from the master but not from the slave.

and

Some of the attachments listed on the slave were showing as attachments linked to draft posts on the local master.

Because the attachments were attached to draft posts, then the logic in get_posts() reported that it did not find the attachment. Consequently another attachment was created. We should try to find a better way to deal with this scenario.

Similarly, I imagine this applies to the parent post. What should happen when a local post is not published?

@bobbingwide

This comment has been minimized.

Copy link
Owner Author

commented Jun 6, 2019

Once all the slave posts have been dealt with we should then turn to all the master posts that have been published but not yet cloned. The question is... do I really want to have the system cloning every published post to the selected slave during reconciliation?

In the case of a set of new block CPTs having been created for a plugin then it would be nice to have the whole lot cloned in one fell swoop, especially if there are a lot of them.
But there may be other content that was created locally that we don’t want to clone.
Do we need to be able to identify in advance whether or not content should be cloned to a particular server? If so, this should be addressed in a separate issue.

@bobbingwide

This comment has been minimized.

Copy link
Owner Author

commented Jun 13, 2019

When there are a lot of posts the echo() method is far too verbose.

Even when verbose=n is set we still get lots of lines starting "None" which makes it harder to see which posts are going to be pushed and which are going to be pulled.
Maybe we should add a terse=y option to further reduce the output.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.