API resquest issues in User Athentication - No acccess to the page to scrap any data

[NicolasFlandrois]

Hi,

From what I saw in my research, Netflix changed its API policy.
It seems to be now covered by User Login Private Authentication.

In short, your code/pip package doesn't work anymore.

The error message:

/<usr/path-to-folder>/netflix/netflix/models.py", line 35, in fetch
    metadata = json.loads(metadata_script_tag.string)
AttributeError: 'NoneType' object has no attribute 'string'

When I pushed my investigations further, it appears that your response object (netflix/models.py line 32) is automatically redirected from the API URL to Netflix Welcome page.

If you unfold your soup = BeautifulSoup(response.content, "html.parser") object (netflix/models.py line 33) in a :

print(soup.text)

your outcome will look like:

Netflix - Watch TV Shows Online, Watch Movies OnlineNetflix and third parties use cookies and similar technologies on this website to collect information about your browsing activities which we use to analyse your use of the website, to personalize our services and to customise our online advertisements. Netflix supports the Digital Advertising Alliance Principles. Learn more about our use of cookies and information. By clicking accept, you accept the use of all cookies and your information for the purposes mentioned above.Netflix and third parties use cookies (why?). You can change (your cookie preferences); by clicking accept, you accept all cookies.AcceptChange your cookie preferencesCloseNetflixSign InUnlimited movies, TV shows, and more.Watch anywhere. Cancel anytime.Get StartedchevronEnjoy on your TV.Watch on Smart TVs, Playstation, Xbox, Chromecast, Apple TV, Blu-ray players, and more.Download your shows to watch offline.Save your favorites easily and always have something to watch.Stranger ThingsDownloading...Watch everywhere.Stream unlimited movies and TV shows on your phone, tablet, laptop, and TV without paying more.Create profiles for kids.Send kids on adventures with their favorite characters in a space made just for them—free with your membership.Frequently Asked QuestionsWhat is Netflix?Netflix is a streaming service that offers a wide variety of award-winning TV shows, movies, anime, documentaries, and more on thousands of internet-connected devices.You can watch as much as you want, whenever you want without a single commercial – all for one low monthly price. There's always something new to discover and new TV shows and movies are added every week!How much does Netflix cost?Watch Netflix on your smartphone, tablet, Smart TV, laptop, or streaming device, all for one fixed monthly fee. Plans range from XXX to XXX a month. No extra costs, no contracts.Where can I watch?Watch anywhere, anytime, on an unlimited number of devices. Sign in with your Netflix account to watch instantly on the web at netflix.com from your personal computer or on any internet-connected device that offers the Netflix app, including smart TVs, smartphones, tablets, streaming media players and game consoles.You can also download your favorite shows with the iOS, Android, or Windows 10 app. Use downloads to watch while you're on the go and without an internet connection. Take Netflix with you anywhere.How do I cancel?Netflix is flexible. There are no pesky contracts and no commitments. You can easily cancel your account online in two clicks. There are no cancellation fees – start or stop your account anytime.What can I watch on Netflix?Netflix has an extensive library of feature films, documentaries, TV shows, anime, award-winning Netflix originals, and more. Watch as much as you want, anytime you want.Is Netflix good for kids?The Netflix Kids experience is included in your membership to give parents control while kids enjoy family-friendly TV shows and movies in their own space.Kids profiles come with PIN-protected parental controls that let you restrict the maturity rating of content kids can watch and block specific titles you don’t want kids to see.Get StartedchevronQuestions? Call (+XXX) XXXX-XXX-XXXXFAQHelp CenterAccountMedia CenterInvestor RelationsJobsRedeem Gift CardsBuy Gift CardsWays to WatchTerms of UsePrivacyCookie PreferencesCorporate InformationContact UsSpeed TestLegal NoticesNetflix OriginalsSelect 

Which text corresponds to the welcome page from Netflix, if not logged in.

I finally could test, with a Netflix account open in a web browser, the issue remains.
As your script/package doesn't use the same authenticated port as the web browser, the connexion isn't recognised. (kinda logic)

---

Apparently, you need to be authenticated, in order to access any data for your BeautifulSoup4 web scrapping.
I would recommend you to have a look to:

https://github.com/jameskang410/scraping-netflix
He handles authentication in his project.

---

On an other note, may I suggest 1 last point.
In order to ease your code review, and debugging, I strongly advise to :

• Extract a requirements.txt file, by doing pip freeze > requirements.txt. Which would ease installing dependency packages.
• Add some documentation/docstring for your Class and Functions, which will help us better understand what/How/Why it all works out. But I cannot cast a stone here, I sometime forget too, and your code is still pretty readable.

---

Thank you to take those requests into considerations.
It's still a neat project!

Have Fun! Stay Safe!