Skip to content
This repository

ADD: No-Op scraper support to allow nfo based only library #1192

Merged
merged 1 commit into from about 1 year ago

5 participants

Chris Browet Matthias Kortstiege jmarshallnz Michael Wehr Martijn Kaijser
Chris Browet
Collaborator
koying commented July 21, 2012

As discussed here: http://forum.xbmc.org/showthread.php?tid=127896&pid=1152628#pid1152628

Allows to have a nfo-based only library.

Matthias Kortstiege
Collaborator
vdrfan commented July 21, 2012

In case other scrapers are enabled you'll still fallback and scrape from the web. You should add a noop check in NfoFile.cpp:104+ to prevent this - in case its the desired behavior?

Mind moving the Scraper.cpp returns after the DEBUG log blocks so we can see what scraper is used in the log?

Chris Browet
Collaborator
koying commented July 22, 2012

Thanks. Updated.

Deleted user
ghost commented July 24, 2012

The content you are editing has changed. Reload the page and try again.

if we absolutely need such a mode it should be done by code, not by a butchered scraper + id checks.

some tick in the content dialog 'fetch online data' or thereabout. if not enabled on the source, scanner handles appropriately.

Sending Request…

Attach images by dragging & dropping or selecting them. Octocat-spinner-32 Uploading your images… Unfortunately, we don't support that file type. Try again with a PNG, GIF, or JPG. Yowza, that's a big file. Try again with an image file smaller than 10MB. This browser doesn't support image attachments. We recommend updating to the latest Internet Explorer, Google Chrome, or Firefox. Something went really wrong, and we can't process that image. Try again.

Chris Browet
Collaborator
koying commented July 24, 2012

That was my first idea, but won't that break the skins (or require updating tall of hem)? I'm not familiar on how UI stuff works in XBMC.

Deleted user
ghost commented July 24, 2012

The content you are editing has changed. Reload the page and try again.

no, the buttons in that dialog are filled from code.

Sending Request…

Attach images by dragging & dropping or selecting them. Octocat-spinner-32 Uploading your images… Unfortunately, we don't support that file type. Try again with a PNG, GIF, or JPG. Yowza, that's a big file. Try again with an image file smaller than 10MB. This browser doesn't support image attachments. We recommend updating to the latest Internet Explorer, Google Chrome, or Firefox. Something went really wrong, and we can't process that image. Try again.

Chris Browet
Collaborator
koying commented July 25, 2012

Not quite, unless I'm mistaken.

CGUIDialogContentSettings defines:

#define CONTROL_CONTENT_TYPE        3
#define CONTROL_SCRAPER_LIST        4
#define CONTROL_SCRAPER_SETTINGS    6

If we want it to do the proper way, "CONTROL_SCRAPER_LIST" & "CONTROL_SCRAPER_SETTINGS" should be under another container which would also contain the "Use local info only" checkbox.
The CONTROL_SCRAPER_... should be disabled if the checkbox is ticked.

So, IMHO, that would require skin rework.

Another solution would be to put the checkbox under CONTROL_SCRAPER_SETTINGS. That would prevent skin issues and make the code prettier, but would be a usability nonsense ;)

Deleted user
ghost commented July 25, 2012

The content you are editing has changed. Reload the page and try again.

Then skin changes it is

Sending Request…

Attach images by dragging & dropping or selecting them. Octocat-spinner-32 Uploading your images… Unfortunately, we don't support that file type. Try again with a PNG, GIF, or JPG. Yowza, that's a big file. Try again with an image file smaller than 10MB. This browser doesn't support image attachments. We recommend updating to the latest Internet Explorer, Google Chrome, or Firefox. Something went really wrong, and we can't process that image. Try again.

Chris Browet
Collaborator
koying commented July 26, 2012

Nevermind.
Scrapers are assumed everywhere so, besides butchering the skins, implementing a "no-scraper" checkbox would take a lot of time, be error-prone and butcher the code for zero functional advantage over my "butchered scraper".

I'd suggest to leave the pull request open so that if some need the functionality, they can use this easily mergeable patch.

jmarshallnz
Owner

You could do it without skin changes by having an internal metadata.null or similar as you've already done, but I don't think you'd need a noop addition to the scraper XML - rather you could just set the library XML to empty and assume XML empty == noop ?

Chris Browet
Collaborator

Sure. Would the patch be deemed acceptable without the "noop" attribute, even withe the "butchered" scrapper?
If so, I'll dig into it.

Michael Wehr
SlrG commented August 25, 2012

+1 for integrating this into XBMC. In my opinion it is a desperately needed feature for people using external scrapers like Ember Media Manager to have full control of the sraping process and make sure only local data is used.

Thank you very much for your work Koying!

@XBMC Team: Please reconsider this pull-request. :)

jmarshallnz
Owner

@koying: IMO if library="" then we can assume the scraper is a noop and other stuff can flow from there, yes.

@cptspiff: is that an acceptable compromise?

Deleted user

The content you are editing has changed. Reload the page and try again.

yes anything without a useless butchered scraper will fly.

Sending Request…

Attach images by dragging & dropping or selecting them. Octocat-spinner-32 Uploading your images… Unfortunately, we don't support that file type. Try again with a PNG, GIF, or JPG. Yowza, that's a big file. Try again with an image file smaller than 10MB. This browser doesn't support image attachments. We recommend updating to the latest Internet Explorer, Google Chrome, or Firefox. Something went really wrong, and we can't process that image. Try again.

Chris Browet
Collaborator

@jmarshallnz I might be wrong, but seeing:

CAddon::CAddon(const AddonProps &props)
  : m_props(props)
  , m_parent(AddonPtr())
{
  if (props.libname.IsEmpty()) BuildLibName();
  else m_strLibName = props.libname;

... I understand that, if the library name is blank, it means use the default name ("default.xml"?), so this couldn't be used to detect an "noop" addon.

@cptspiff
Would

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<scraper framework="1.1">
</scraper>

be acceptable?
IMO, in this instance, an empty scrapper is not "butchered useless". It is just a scrapper doing nothing, which is what we seek and a valid case.
The surrounding code is just a mean to handle this case of empty scrapper.

Chris Browet
Collaborator

@jmarshallnz @cptspiff Is it ok for merge in the current state?

jmarshallnz
Owner

I guess the only thing I don't like about it is that it's a special case (i.e. we have to remember to take care of IsNoop() in a bunch of places). How many of those are actually required - i.e. what happens currently with a scraper without CreateSearchURL() et. al. ?

Hmm, maybe it's just the naming I don't like (maybe scraper->CanScrape()?) ?

Chris Browet
Collaborator

IIRC, without the noop(), the import fails and nothing is imported, which is logical.
There was a loophole at one point, because a badly written scrapper I used added "empty" items even if it was not found in its database.
Would you prefer something like that? Seems even dirtier to me ;)

jmarshallnz
Owner

I think the key is to have as little special-casing as possible. If what you have is minimal, then that's fine.

Martijn Kaijser
Owner

@koying
please rebase

Chris Browet koying merged commit 9fdb68a into from April 01, 2013
Chris Browet koying closed this April 01, 2013
Chris Browet koying deleted the branch April 01, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 1 unique commit by 1 author.

Apr 01, 2013
Chris Browet ADD: No-Op scraper support to allow nfo based only library 5527a6b
This page is out of date. Refresh to see the latest.
29  addons/metadata.local/addon.xml
... ...
@@ -0,0 +1,29 @@
  1
+<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
  2
+<addon id="metadata.local"
  3
+       name="Local information only"
  4
+       version="1.0.0"
  5
+       provider-name="Team XBMC">
  6
+  <requires>
  7
+    <import addon="xbmc.metadata" version="1.0"/>
  8
+  </requires>
  9
+  <extension point="xbmc.metadata.scraper.albums"
  10
+             language="multi"
  11
+             library="local.xml"/>
  12
+  <extension point="xbmc.metadata.scraper.artists"
  13
+             language="multi"
  14
+             library="local.xml"/>
  15
+  <extension point="xbmc.metadata.scraper.musicvideos"
  16
+             language="multi"
  17
+             library="local.xml"/>
  18
+  <extension point="xbmc.metadata.scraper.tvshows"
  19
+             language="multi"
  20
+             library="local.xml"/>
  21
+  <extension point="xbmc.metadata.scraper.movies"
  22
+             language="multi"
  23
+             library="local.xml"/>
  24
+  <extension point="xbmc.addon.metadata">
  25
+    <summary lang="en">Local Infomation only pseudo-scraper</summary>
  26
+    <description lang="en">Use local information only</description>
  27
+    <platform>all</platform>
  28
+  </extension>
  29
+</addon>
3  addons/metadata.local/local.xml
... ...
@@ -0,0 +1,3 @@
  1
+<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
  2
+<scraper framework="1.1">
  3
+</scraper>
5  xbmc/NfoFile.cpp
@@ -131,6 +131,11 @@ CNfoFile::NFOResult CNfoFile::Create(const CStdString& strPath, const ScraperPtr
131 131
 // return value: 0 - success; 1 - no result; skip; 2 - error
132 132
 int CNfoFile::Scrape(ScraperPtr& scraper)
133 133
 {
  134
+  if (scraper->IsNoop())
  135
+  {
  136
+    m_scurl = CScraperUrl();
  137
+    return 0;
  138
+  }
134 139
   if (scraper->Type() != m_type)
135 140
     return 1;
136 141
   scraper->ClearCache();
32  xbmc/addons/Scraper.cpp
@@ -412,12 +412,23 @@ bool CScraper::IsInUse() const
412 412
   return false;
413 413
 }
414 414
 
  415
+bool CScraper::IsNoop()
  416
+{
  417
+    if (!Load())
  418
+      throw CScraperError();
  419
+
  420
+    return m_parser.IsNoop();
  421
+}
  422
+
415 423
 // pass in contents of .nfo file; returns URL (possibly empty if none found)
416 424
 // and may populate strId, or throws CScraperError on error
417 425
 CScraperUrl CScraper::NfoUrl(const CStdString &sNfoContent)
418 426
 {
419 427
   CScraperUrl scurlRet;
420 428
 
  429
+  if (IsNoop())
  430
+    return scurlRet;
  431
+
421 432
   // scraper function takes contents of .nfo file, returns XML (see below)
422 433
   vector<CStdString> vcsIn;
423 434
   vcsIn.push_back(sNfoContent);
@@ -490,14 +501,18 @@ std::vector<CScraperUrl> CScraper::FindMovie(XFILE::CCurlFile &fcurl, const CStd
490 501
   CStdString sTitle, sTitleYear, sYear;
491 502
   CUtil::CleanString(sMovie, sTitle, sTitleYear, sYear, true/*fRemoveExt*/, fFirst);
492 503
 
493  
-  if (!fFirst || Content() == CONTENT_MUSICVIDEOS)
494  
-    sTitle.Replace("-"," ");
495  
-
496 504
   CLog::Log(LOGDEBUG, "%s: Searching for '%s' using %s scraper "
497 505
     "(path: '%s', content: '%s', version: '%s')", __FUNCTION__, sTitle.c_str(),
498 506
     Name().c_str(), Path().c_str(),
499 507
     ADDON::TranslateContent(Content()).c_str(), Version().c_str());
500 508
 
  509
+  std::vector<CScraperUrl> vcscurl;
  510
+  if (IsNoop())
  511
+    return vcscurl;
  512
+
  513
+  if (!fFirst || Content() == CONTENT_MUSICVIDEOS)
  514
+    sTitle.Replace("-"," ");
  515
+
501 516
   sTitle.ToLower();
502 517
 
503 518
   vector<CStdString> vcsIn(1);
@@ -509,7 +524,6 @@ std::vector<CScraperUrl> CScraper::FindMovie(XFILE::CCurlFile &fcurl, const CStd
509 524
   // request a search URL from the title/filename/etc.
510 525
   CScraperUrl scurl;
511 526
   vector<CStdString> vcsOut = Run("CreateSearchUrl", scurl, fcurl, &vcsIn);
512  
-  std::vector<CScraperUrl> vcscurl;
513 527
   if (vcsOut.empty())
514 528
   {
515 529
     CLog::Log(LOGDEBUG, "%s: CreateSearchUrl failed", __FUNCTION__);
@@ -616,6 +630,10 @@ std::vector<CMusicAlbumInfo> CScraper::FindAlbum(CCurlFile &fcurl, const CStdStr
616 630
     sAlbum.c_str(), Name().c_str(), Path().c_str(),
617 631
     ADDON::TranslateContent(Content()).c_str(), Version().c_str());
618 632
 
  633
+  std::vector<CMusicAlbumInfo> vcali;
  634
+  if (IsNoop())
  635
+    return vcali;
  636
+
619 637
   // scraper function is given the album and artist as parameters and
620 638
   // returns an XML <url> element parseable by CScraperUrl
621 639
   std::vector<CStdString> extras(2);
@@ -628,7 +646,6 @@ std::vector<CMusicAlbumInfo> CScraper::FindAlbum(CCurlFile &fcurl, const CStdStr
628 646
   if (vcsOut.size() > 1)
629 647
     CLog::Log(LOGWARNING, "%s: scraper returned multiple results; using first", __FUNCTION__);
630 648
 
631  
-  std::vector<CMusicAlbumInfo> vcali;
632 649
   if (vcsOut.empty() || vcsOut[0].empty())
633 650
     return vcali;
634 651
   scurl.ParseString(vcsOut[0]);
@@ -710,6 +727,10 @@ std::vector<CMusicArtistInfo> CScraper::FindArtist(CCurlFile &fcurl,
710 727
     Name().c_str(), Path().c_str(),
711 728
     ADDON::TranslateContent(Content()).c_str(), Version().c_str());
712 729
 
  730
+  std::vector<CMusicArtistInfo> vcari;
  731
+  if (IsNoop())
  732
+    return vcari;
  733
+
713 734
   // scraper function is given the artist as parameter and
714 735
   // returns an XML <url> element parseable by CScraperUrl
715 736
   std::vector<CStdString> extras(1);
@@ -718,7 +739,6 @@ std::vector<CMusicArtistInfo> CScraper::FindArtist(CCurlFile &fcurl,
718 739
   CScraperUrl scurl;
719 740
   vector<CStdString> vcsOut = RunNoThrow("CreateArtistSearchUrl", scurl, fcurl, &extras);
720 741
 
721  
-  std::vector<CMusicArtistInfo> vcari;
722 742
   if (vcsOut.empty() || vcsOut[0].empty())
723 743
     return vcari;
724 744
   scurl.ParseString(vcsOut[0]);
1  xbmc/addons/Scraper.h
@@ -116,6 +116,7 @@ class CScraper : public CAddon
116 116
   bool Supports(const CONTENT_TYPE &content) const;
117 117
 
118 118
   bool IsInUse() const;
  119
+  bool IsNoop();
119 120
 
120 121
   // scraper media functions
121 122
   CScraperUrl NfoUrl(const CStdString &sNfoContent);
5  xbmc/utils/ScraperParser.cpp
@@ -42,6 +42,7 @@ CScraperParser::CScraperParser()
42 42
   m_document = NULL;
43 43
   m_SearchStringEncoding = "UTF-8";
44 44
   m_scraper = NULL;
  45
+  m_isNoop = true;
45 46
 }
46 47
 
47 48
 CScraperParser::CScraperParser(const CScraperParser& parser)
@@ -50,6 +51,7 @@ CScraperParser::CScraperParser(const CScraperParser& parser)
50 51
   m_document = NULL;
51 52
   m_SearchStringEncoding = "UTF-8";
52 53
   m_scraper = NULL;
  54
+  m_isNoop = true;
53 55
   *this = parser;
54 56
 }
55 57
 
@@ -115,6 +117,7 @@ bool CScraperParser::LoadFromXML()
115 117
     TiXmlElement* pChildElement = m_pRootElement->FirstChildElement("CreateSearchUrl");
116 118
     if (pChildElement)
117 119
     {
  120
+      m_isNoop = false;
118 121
       if (!(m_SearchStringEncoding = pChildElement->Attribute("SearchStringEncoding")))
119 122
         m_SearchStringEncoding = "UTF-8";
120 123
     }
@@ -122,12 +125,14 @@ bool CScraperParser::LoadFromXML()
122 125
     pChildElement = m_pRootElement->FirstChildElement("CreateArtistSearchUrl");
123 126
     if (pChildElement)
124 127
     {
  128
+      m_isNoop = false;
125 129
       if (!(m_SearchStringEncoding = pChildElement->Attribute("SearchStringEncoding")))
126 130
         m_SearchStringEncoding = "UTF-8";
127 131
     }
128 132
     pChildElement = m_pRootElement->FirstChildElement("CreateAlbumSearchUrl");
129 133
     if (pChildElement)
130 134
     {
  135
+      m_isNoop = false;
131 136
       if (!(m_SearchStringEncoding = pChildElement->Attribute("SearchStringEncoding")))
132 137
         m_SearchStringEncoding = "UTF-8";
133 138
     }
2  xbmc/utils/ScraperParser.h
@@ -45,6 +45,7 @@ class CScraperParser
45 45
   ~CScraperParser();
46 46
   CScraperParser& operator= (const CScraperParser& parser);
47 47
   bool Load(const CStdString& strXMLFile);
  48
+  bool IsNoop() { return m_isNoop; };
48 49
 
49 50
   void Clear();
50 51
   const CStdString GetFilename() { return m_strFile; }
@@ -76,6 +77,7 @@ class CScraperParser
76 77
   TiXmlElement* m_pRootElement;
77 78
 
78 79
   const char* m_SearchStringEncoding;
  80
+  bool m_isNoop;
79 81
 
80 82
   CStdString m_strFile;
81 83
   ADDON::CScraper* m_scraper;
2  xbmc/video/windows/GUIWindowVideoBase.cpp
@@ -511,7 +511,7 @@ bool CGUIWindowVideoBase::ShowIMDB(CFileItem *item, const ScraperPtr &info2)
511 511
       if (needsRefresh)
512 512
       {
513 513
         bHasInfo = true;
514  
-        if (nfoResult == CNfoFile::URL_NFO || nfoResult == CNfoFile::COMBINED_NFO || nfoResult == CNfoFile::FULL_NFO)
  514
+        if (!info->IsNoop() && (nfoResult == CNfoFile::URL_NFO || nfoResult == CNfoFile::COMBINED_NFO || nfoResult == CNfoFile::FULL_NFO))
515 515
         {
516 516
           if (CGUIDialogYesNo::ShowAndGetInput(13346,20446,20447,20022))
517 517
           {
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.