AttributeError: 'NoneType' object has no attribute 'find'

[hasamba]

Hi,
this is the output im getting

hasamba@HasambaMint1:~/Apps/RatS$ python3 transfer_ratings.py --source trakt --destination imdb
===== Trakt: Parsing 10 pages with 551 movies in total
 15% (84 of 551) |################                                                                                           | Elapsed Time: 0:08:10 ETA:   0:52:01Traceback (most recent call last):
  File "transfer_ratings.py", line 230, in <module>
    main()
  File "transfer_ratings.py", line 82, in main
    execute(args)
  File "transfer_ratings.py", line 131, in execute
    movies = execute_parsing(args, parser)
  File "transfer_ratings.py", line 174, in execute_parsing
    movies = parse_data_from_source(parser)
  File "transfer_ratings.py", line 180, in parse_data_from_source
    movies = parser.parse()
  File "/home/hasamba/Apps/RatS/RatS/base/base_ratings_parser.py", line 37, in parse
    raise e
  File "/home/hasamba/Apps/RatS/RatS/base/base_ratings_parser.py", line 33, in parse
    self._parse_ratings()
  File "/home/hasamba/Apps/RatS/RatS/base/base_ratings_parser.py", line 48, in _parse_ratings
    pages_count = self._retrieve_pages_count_and_movies_count(movie_ratings_page)
  File "/home/hasamba/Apps/RatS/RatS/base/base_ratings_parser.py", line 79, in _retrieve_pages_count_and_movies_count
    self.movies_count = self._get_movies_count(movie_ratings_page)
  File "/home/hasamba/Apps/RatS/RatS/trakt/trakt_ratings_parser.py", line 16, in _get_movies_count
    return int(movie_ratings_page.find('section', class_='subnav-wrapper').
AttributeError: 'NoneType' object has no attribute 'find'

[OdinGitDat]

I have the same with source imdb and destination trakt

When importing from imdb.

• Version: commit 6a6c9c3
• Platform OS: UbuntuLTS 18
• Running in a virtualenv?: LXC
• Running in Docker?: yes
• Executed command: docker run --rm -it -v ~/.RatS.cfg:/RatS/RatS/credentials.cfg stegschreck/rats:master python3 transfer_ratings.py --source imdb --destination trakt

Stacktrace

===== IMDB: CSV downloaded to /RatS/RatS/exports/20201011154942_IMDB.csv
===== getting movies from CSV

===== IMDB: saved 605 parsed movies to /RatS/RatS/exports/20201011154942_IMDB.json
===== Trakt: posting 605 movies
  0% (6 of 605) |                                                                | Elapsed Time: 0:00:51 ETA:   1:23:50
ERROR: There was an exception inside Trakt (see below). Skipping insertion.

===== ABORTING =====
  0% (6 of 605) |                                                                | Elapsed Time: 0:00:56 ETA:   2:13:57Traceback (most recent call last):
  File "transfer_ratings.py", line 212, in insert_movie_ratings
    inserter.insert(movies, source)
  File "/RatS/RatS/base/base_ratings_inserter.py", line 42, in insert
    movie_details_page_found = self._go_to_movie_details_page(movie)
  File "/RatS/RatS/base/base_ratings_inserter.py", line 96, in _go_to_movie_details_page
    success = self._find_movie(movie)
  File "/RatS/RatS/base/base_ratings_inserter.py", line 118, in _find_movie
    return self._is_movie_in_search_results(movie, search_results)
  File "/RatS/RatS/base/base_ratings_inserter.py", line 122, in _is_movie_in_search_results
    if self._is_requested_movie(movie, search_result):
  File "/RatS/RatS/trakt/trakt_ratings_inserter.py", line 29, in _is_requested_movie
    return self._check_movie_details(movie, search_result)
  File "/RatS/RatS/trakt/trakt_ratings_inserter.py", line 40, in _check_movie_details
    return self._compare_external_links(self.site.browser.page_source, movie, 'imdb.com', 'imdb')
  File "/RatS/RatS/trakt/trakt_ratings_inserter.py", line 50, in _compare_external_links
    external_links = movie_details_page.find(id='info-wrapper').find('ul', class_='external').find_all('a')
AttributeError: 'NoneType' object has no attribute 'find'

===== IMDB: CSV downloaded to /RatS/RatS/exports/20201011161247_IMDB.csv
===== getting movies from CSV

===== IMDB: saved 605 parsed movies to /RatS/RatS/exports/20201011161247_IMDB.json
===== Trakt: posting 605 movies
  1% (10 of 605) |#                                                              | Elapsed Time: 0:01:22 ETA:   1:32:03
ERROR: There was an exception inside Trakt (see below). Skipping insertion.

===== ABORTING =====
  1% (10 of 605) |#                                                              | Elapsed Time: 0:01:27 ETA:   2:24:10Traceback (most recent call last):
  File "transfer_ratings.py", line 212, in insert_movie_ratings
    inserter.insert(movies, source)
  File "/RatS/RatS/base/base_ratings_inserter.py", line 42, in insert
    movie_details_page_found = self._go_to_movie_details_page(movie)
  File "/RatS/RatS/base/base_ratings_inserter.py", line 96, in _go_to_movie_details_page
    success = self._find_movie(movie)
  File "/RatS/RatS/base/base_ratings_inserter.py", line 118, in _find_movie
    return self._is_movie_in_search_results(movie, search_results)
  File "/RatS/RatS/base/base_ratings_inserter.py", line 122, in _is_movie_in_search_results
    if self._is_requested_movie(movie, search_result):
  File "/RatS/RatS/trakt/trakt_ratings_inserter.py", line 29, in _is_requested_movie
    return self._check_movie_details(movie, search_result)
  File "/RatS/RatS/trakt/trakt_ratings_inserter.py", line 40, in _check_movie_details
    return self._compare_external_links(self.site.browser.page_source, movie, 'imdb.com', 'imdb')
  File "/RatS/RatS/trakt/trakt_ratings_inserter.py", line 50, in _compare_external_links
    external_links = movie_details_page.find(id='info-wrapper').find('ul', class_='external').find_all('a')
AttributeError: 'NoneType' object has no attribute 'find'

[StegSchreck]

After investigating a bit I found other examples that might be connected.

Traceback (most recent call last):
  File "transfer_ratings.py", line 234, in <module>
    main()
  File "transfer_ratings.py", line 86, in main
    execute(args)
  File "transfer_ratings.py", line 135, in execute
    movies = execute_parsing(args, parser)
  File "transfer_ratings.py", line 178, in execute_parsing
    movies = parse_data_from_source(parser)
  File "transfer_ratings.py", line 184, in parse_data_from_source
    movies = parser.parse()
  File "/home/sebastian/repos/RatS/RatS/base/base_ratings_parser.py", line 37, in parse
    raise e
  File "/home/sebastian/repos/RatS/RatS/base/base_ratings_parser.py", line 33, in parse
    self._parse_ratings()
  File "/home/sebastian/repos/RatS/RatS/base/base_ratings_parser.py", line 75, in _parse_ratings
    self._parse_movie_listing_page(movie_listing_page)
  File "/home/sebastian/repos/RatS/RatS/base/base_ratings_parser.py", line 96, in _parse_movie_listing_page
    movie = self._parse_movie_tile(movie_tile)
  File "/home/sebastian/repos/RatS/RatS/base/base_ratings_parser.py", line 155, in _parse_movie_tile
    raise e
  File "/home/sebastian/repos/RatS/RatS/base/base_ratings_parser.py", line 151, in _parse_movie_tile
    self.parse_movie_details_page(movie)
  File "/home/sebastian/repos/RatS/RatS/trakt/trakt_ratings_parser.py", line 46, in parse_movie_details_page
    movie_year = movie_details_page.find(class_='year').get_text()
AttributeError: 'NoneType' object has no attribute 'get_text'

Traceback (most recent call last):
  File "transfer_ratings.py", line 234, in <module>
    main()
  File "transfer_ratings.py", line 86, in main
    execute(args)
  File "transfer_ratings.py", line 135, in execute
    movies = execute_parsing(args, parser)
  File "transfer_ratings.py", line 178, in execute_parsing
    movies = parse_data_from_source(parser)
  File "transfer_ratings.py", line 184, in parse_data_from_source
    movies = parser.parse()
  File "/home/sebastian/repos/RatS/RatS/base/base_ratings_parser.py", line 37, in parse
    raise e
  File "/home/sebastian/repos/RatS/RatS/base/base_ratings_parser.py", line 33, in parse
    self._parse_ratings()
  File "/home/sebastian/repos/RatS/RatS/base/base_ratings_parser.py", line 48, in _parse_ratings
    pages_count = self._retrieve_pages_count_and_movies_count(movie_ratings_page)
  File "/home/sebastian/repos/RatS/RatS/base/base_ratings_parser.py", line 79, in _retrieve_pages_count_and_movies_count
    self.movies_count = self._get_movies_count(movie_ratings_page)
  File "/home/sebastian/repos/RatS/RatS/trakt/trakt_ratings_parser.py", line 17, in _get_movies_count
    return int(movie_ratings_page.find('section', class_='subnav-wrapper').
AttributeError: 'NoneType' object has no attribute 'find'

These were caused by Trakt (or rather the Cloudflare CDN they are using) responding with an HTTP code 521. Meaning that Cloudflare could not connect to the Trakt backend. See http://getstatuscode.com/521