Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Orion 15.2.0 psvamb 6815 #8852

Merged
merged 5 commits into from Nov 1, 2019
Merged
Changes from 1 commit
Commits
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

Prev

PSVAMB-6815 - code review

  • Loading branch information
hilak committed Nov 1, 2019
commit 4bf1d46c6c1a241babfa6bd42a191642e4c193f7
@@ -10,6 +10,8 @@ class KApFeedDropFolderEngine extends KFeedDropFolderEngine
*/
protected $apiKey;

const AP_FEED_PAGE_SIZE = 100;

public function watchFolder(KalturaDropFolder $dropFolder)
{
/* @var $dropFolder KalturaApFeedDropFolder */
@@ -29,6 +31,11 @@ public function watchFolder(KalturaDropFolder $dropFolder)
//Get Drop Folder feed and import it into an array
$feedContent = $this->fetchFeedContent($feedUrl);
$feed = json_decode($feedContent, true);
if (!$feed)
{
KalturaLog::info('Feed page could not be displayed, no way to continue. Breaking out of the process.');
break;
}
$lastPageCount = $feed['data']['current_item_count'];

$feedItems = $feed['data']['items'];
@@ -45,6 +52,11 @@ public function watchFolder(KalturaDropFolder $dropFolder)

$datum = $part['item'];
$feedItemJson = json_decode($this->fetchFeedContent($datum['uri']), true);
if (!$feedItemJson)
{
KalturaLog::info('Feed item could not be retrieved, continue to the next one.');
continue;
}

$feedItem = new DOMDocument();
$feedItem->loadXML('<item/>');
@@ -53,7 +65,7 @@ public function watchFolder(KalturaDropFolder $dropFolder)

KalturaLog::info ('Single item: ' . print_r($feedItem->saveXML(), true));

$counter += $this->watchProcessSingleItem(simplexml_import_dom($feedItem), $existingDropFolderFilesMap, $counter);
$counter += $this->watchProcessSingleItem(simplexml_import_dom($feedItem), $existingDropFolderFilesMap);

}

@@ -68,7 +80,7 @@ public function watchFolder(KalturaDropFolder $dropFolder)
$feedUrl = $feed['data']['next_page'];
}

}while ($lastPageCount == 100);
}while ($lastPageCount == self::AP_FEED_PAGE_SIZE);

This comment has been minimized.

Copy link
@MosheMaorKaltura

MosheMaorKaltura Nov 3, 2019

Contributor

This is a weak end case, since it is possible that due to changes that happen in parallel, one of the pages will return less than page size, still there will be more to come.
Safer approach will be until lastPageCount == 0.

This comment has been minimized.

Copy link
@hilak

hilak Nov 4, 2019

Author Contributor

@MosheMaorKaltura because this is a professionally generated news feed, odds are slim that we could catch it just at the time of day it gets updated, which is exactly why I cut off the interaction with it on what is perceived as the last page. Also I don't see how the same case can't happen even if the last page item count is 0. This interaction isn't a one-off - this is a drop folder pull, so it gets repeated periodically. Any content that was not ingested on a single run will likely get pulled in on the next one.


foreach ($existingDropFolderFilesMap as $existingDropFolderFile)
{
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.