Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixed bridge #1195

Merged
merged 9 commits into from
Jun 28, 2019
Merged

Fixed bridge #1195

Changes from 3 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
1f50df4
Fixed bridge by changing the way the articles are loaded AND their ti…
Riduidel Jun 28, 2019
d9877b3
Fixing linter violations
Riduidel Jun 28, 2019
225e2fd
screw my bad IDE config !
Riduidel Jun 28, 2019
9d72747
Update bridges/GQMagazineBridge.php
Riduidel Jun 28, 2019
0c7a8a5
Update bridges/GQMagazineBridge.php
Riduidel Jun 28, 2019
e19898d
Update bridges/GQMagazineBridge.php
Riduidel Jun 28, 2019
3f4913b
Update bridges/GQMagazineBridge.php
Riduidel Jun 28, 2019
7e86f2c
replacing an if by a switch should be better performance-wise
Riduidel Jun 28, 2019
6bbd319
Damn, I failed at tabs !
Riduidel Jun 28, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
64 changes: 36 additions & 28 deletions bridges/GQMagazineBridge.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,11 @@ class GQMagazineBridge extends BridgeAbstract
'data-original' => 'src'
);

const POSSIBLE_TITLES = array(
'h2',
'h3'
);

private function getDomain() {
$domain = $this->getInput('domain');
if (empty($domain))
Expand All @@ -54,6 +59,17 @@ public function getURI()
return $this->getDomain() . '/' . $this->getInput('page');
}

private function findTitleOf($link) {
foreach (self::POSSIBLE_TITLES as $tag) {
$title = $link->find($tag, 0);
if($title != null) {
Riduidel marked this conversation as resolved.
Show resolved Hide resolved
if($title->plaintext != null) {
Riduidel marked this conversation as resolved.
Show resolved Hide resolved
return $title->plaintext;
}
}
}
}

public function collectData()
{
$html = getSimpleHTMLDOM($this->getURI()) or returnServerError('Could not request ' . $this->getURI());
Expand All @@ -62,30 +78,31 @@ public function collectData()
$main = $html->find('main', 0);
foreach ($main->find('a') as $link) {
$uri = $link->href;
$title = $link->find('h2', 0);
$date = $link->find('time', 0);

$item = array();
$author = $link->find('span[itemprop=name]', 0);
$item['author'] = $author->plaintext;
$item['title'] = $title->plaintext;
if(substr($uri, 0, 1) === 'h') { // absolute uri
$item['uri'] = $uri;
} else if(substr($uri, 0, 1) === '/') { // domain relative url
$item['uri'] = $this->getDomain() . $uri;
} else {
$item['uri'] = $this->getDomain() . '/' . $uri;
}
if($author != null) {
Riduidel marked this conversation as resolved.
Show resolved Hide resolved
$item['author'] = $author->plaintext;
$item['title'] = $this->findTitleOf($link);
if(substr($uri, 0, 1) === 'h') { // absolute uri
Riduidel marked this conversation as resolved.
Show resolved Hide resolved
$item['uri'] = $uri;
} else if(substr($uri, 0, 1) === '/') { // domain relative url
Riduidel marked this conversation as resolved.
Show resolved Hide resolved
$item['uri'] = $this->getDomain() . $uri;
} else {
$item['uri'] = $this->getDomain() . '/' . $uri;
}

$article = $this->loadFullArticle($item['uri']);
if($article) {
$item['content'] = $this->replaceUriInHtmlElement($article);
} else {
$item['content'] = "<strong>Article body couldn't be loaded</strong>. It must be a bug!";
$article = $this->loadFullArticle($item['uri']);
if($article) {
$item['content'] = $this->replaceUriInHtmlElement($article);
} else {
$item['content'] = "<strong>Article body couldn't be loaded</strong>. It must be a bug!";
}
$short_date = $date->datetime;
$item['timestamp'] = strtotime($short_date);
$this->items[] = $item;
}
$short_date = $date->datetime;
$item['timestamp'] = strtotime($short_date);
$this->items[] = $item;
}
}

Expand All @@ -96,16 +113,7 @@ public function collectData()
*/
private function loadFullArticle($uri){
$html = getSimpleHTMLDOMCached($uri);
// Once again, that generated css classes madness is an obstacle ... which i can go over easily
foreach($html->find('div') as $div) {
// List the CSS classes of that div
$classes = $div->class;
// I can't directly lookup that class since GQ since to generate random names like "ArticleBodySection-fkggUW"
if(strpos($classes, 'ArticleBodySection') !== false) {
return $div;
}
}
return null;
return $html->find('section[data-test-id=ArticleBodyContent]', 0);
}

/**
Expand Down