Tracking original IDs or general order

[pavelevap]

During work on test infrastructure we found a problem with order of posts on cloned site. When there are several posts (or other items) published at the same time and ordered by that time, their order can be different on original and cloned site, see example in #1409 (comment).

Order on original site is based on ID (order from database storage), but cloned site depends on loading .ini files through iterators:

versionpress/plugins/versionpress/src/Storages/DirectoryStorage.php

Lines 157 to 168 in e220983

	private function getEntityFiles()
	{
	if (!is_dir($this->directory)) {
	return [];
	}
	$directoryIterator = new RecursiveDirectoryIterator($this->directory);
	$recursiveIterator = new RecursiveIteratorIterator($directoryIterator);
	$iniFilesIterator = new RegexIterator($recursiveIterator, '~^.+\.ini$~i', RecursiveRegexIterator::GET_MATCH);
	return array_keys(iterator_to_array($iniFilesIterator));
	}

And VersionPress ignores original IDs, so there is currently no way to order posts by their original ID on cloned sites.

Maybe there is a time to start tracking original IDs (VP could make a clean clone with original IDs which could be helpful in some cases) or general order (VP could generally know order of created items at least).

[borekb]

We spent some time chatting about this with @JanVoracek, nothing too deep or well-explored yet but to record the key points:

• Storing auto-increment IDs is probably not doable. Any re-numbering of IDs would lead to changes in the vpdb files, leading to new commits, leading to strange Git history at the very least.
• Order represented by precise time of creation is probably doable. (I'll be calling it 'timestamps' below but no specific implementation is assumed yet.)
• There are generally two places to store the timestamp: file name (path) or file contents.
  • Path storage should be faster to work with as there would be no need to read contents of files.
    • Prefixing VPID with timestamp or incorporating it into the beginning of it would have a problem that files would not be evenly distributed to 256 folders. (VersionPress uses first two letters of VPID as a subfolder under vpdb/<entity> to deal with situations where thousands or even millions of files would end up in a single folder.) So it would still be some challenge to order by timestamp correctly but generally, it should be more efficient than reading files contents.
  • Storing timestemp in file contents feels a bit more flexible, maybe we'd even need it for things like postmeta. But also possibly inefficient.
    • Performance is an important concern as, for example, restoring site from Git or creating a staging environment will need to take all timestamps into account. In the worst case, this means parsing a lot of files and building a large data structure in memory.
• We acknowledge this is an issue that should be solved one day but don't see it as a high priority. There haven't been reports about this yet in the support repo or other channels.