Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Instagram changed its inner data structure #145

Closed
lynxrv21 opened this issue Mar 13, 2018 · 29 comments

Comments

@lynxrv21
Copy link

commented Mar 13, 2018

Getting the same error for any username I try.

instaLooter instagram
00:41:53 - WARNING - Could not find page of user: instagram
00:41:53 - WARNING - No medias found.
@bilyboy785

This comment has been minimized.

Copy link

commented Mar 14, 2018

+1

@bilyboy785

This comment has been minimized.

Copy link

commented Mar 14, 2018

Do you think fix this or it is normal ?

@jnskw

This comment has been minimized.

Copy link

commented Mar 14, 2018

Instagram might have changed the structure of its data.

I think Instagram-Scraper had a similar Problem:
rarcega/instagram-scraper@3840aab

@lynxrv21

This comment has been minimized.

Copy link
Author

commented Mar 14, 2018

So, is anyone going to fix it here, or I have to do it by myself? :)

@bilyboy785

This comment has been minimized.

Copy link

commented Mar 14, 2018

if you fix it, can you share it ? :)

@lynxrv21

This comment has been minimized.

Copy link
Author

commented Mar 14, 2018

No luck with this yet.
Looks like we need @althonos to help us.

@bilyboy785

This comment has been minimized.

Copy link

commented Mar 15, 2018

Only he does not look active anymore.

@althonos

This comment has been minimized.

Copy link
Owner

commented Mar 15, 2018

My studies are calling, so I don't have much time to invest in this.
The main issue is that IG changed not only its internal data representation, but also the method used to query those data. This is what makes fixing a pain. I'll try to see what I can do.

@AbysmalBiscuit

This comment has been minimized.

Copy link
Contributor

commented Mar 15, 2018

Hey,

I've had a look at the new json structure, and as @althonos says it's different from what instaLooter expects to find, hence why there is an internal KeyError when the response data gets processed for the firt time in core.py by:
media_info = data['entry_data'][self._page_name][0][self._section_name]['media']

From analyzing the json reponse, all the information that may be wanted is in there: image src url, metadata etc...

This is type of code that's needed to get what I think should be all the media info that instaLooter expects ( this will just print the response contents of all posts loaded in the first response):
https://gist.github.com/AbysmalBiscuit/0799d5ed77e9ec6415836d8c8da9fd2d

@althonos

This comment has been minimized.

Copy link
Owner

commented Mar 15, 2018

I've got a working-ish fix on the dev branch you can install with:

pip install --user --upgrade https://github.com/althonos/InstaLooter/archive/6d6dac1.zip

I still want to check the code in greater detail before pushing a release, but at least it downloads pics as intended for both users and hashtags.

@althonos althonos changed the title 'Could not find page or user' error for any user. Instagram changed its inner data structure Mar 15, 2018

@althonos althonos added the bug label Mar 15, 2018

@AbysmalBiscuit

This comment has been minimized.

Copy link
Contributor

commented Mar 15, 2018

@althonos Thanks for the quick fix!

Edit: Lines 339 and 340 have a pprint call:
from pprint import pprint
pprint(data['data'][self._section_name].keys())
that prints dict_keys(['edge_owner_to_timeline_media']) with every page that gets loaded, but otherwise everything seems to work great. :)

@althonos

This comment has been minimized.

Copy link
Owner

commented Mar 15, 2018

@AbysmalBiscuit : fixed, thanks.

@oferchen

This comment has been minimized.

Copy link

commented Mar 17, 2018

another issue could be related,
Exception in thread Thread-5:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/local/lib/python2.7/dist-packages/instaLooter/worker.py", line 70, in run
self._download_photo(media)
File "/usr/local/lib/python2.7/dist-packages/instaLooter/worker.py", line 133, in _download_photo
self._add_metadata(photo_name, media)
File "/usr/local/lib/python2.7/dist-packages/instaLooter/worker.py", line 106, in _add_metadata
self._get_caption(metadata).encode('utf-8'),
File "/usr/local/lib/python2.7/dist-packages/instaLooter/worker.py", line 45, in _get_caption
return metadata['edge_media_to_caption']['edges'][0]['node']['text']
IndexError: list index out of range

@JonasGrobe

This comment has been minimized.

Copy link

commented Mar 18, 2018

Using timeframes still doesn't work - had to change line 416 in core.py from "for media in page[media]['edges']:" to "for media in page[medias]['edges']:"

@Bigben83

This comment has been minimized.

Copy link

commented Mar 19, 2018

I pulled the dev branch and the could not find page of user is still showing ? should i be doing something else ?

@althonos

This comment has been minimized.

Copy link
Owner

commented Mar 19, 2018

@Bigben83 : could you give me some parameters so that I can reproduce this ? It works just fine with a public user on my machine.

@Bigben83

This comment has been minimized.

Copy link

commented Mar 19, 2018

@althonos this is what i have been using
/usr/local/bin/instaLooter sawyerhartman /home/dev/public_html/instagram/sawyerhartman -T {username}.{datetime} -q --add-metadata -v -N -n 50

@rajnishcseku

This comment has been minimized.

Copy link

commented Mar 21, 2018

@althonos I am also having the same issue which @Bigben83 is having.

@althonos

This comment has been minimized.

Copy link
Owner

commented Mar 21, 2018

@Bigben83 @rajnishcseku @JonasGrobe @oferchen @AbysmalBiscuit @jnskw @lynxrv21 @bilyboy785 : If you want to test the new version, you can pull from the dev branch.

The codebase is brand new, but the CLI should be very familiar, and most problems should be fixed. Note that I didn't test the login at all since I have no IG account and I don't know if the login procedure used before still works !

I'll release to PyPI once I've got the documentation updated and the API stabilised.

In the meantime:

pip install --user -U https://github.com/althonos/InstaLooter/archive/dev.zip

Oh, and by the way, you cannot directly download a profile without using the instalooter user subcommand. This is easier for both docopt and myself 😉

@rajnishcseku

This comment has been minimized.

Copy link

commented Mar 21, 2018

@althonos

Hi, I followed following steps:

  1. pip uninstall instaLooter

Output:

C:\Users\rajnish.rana\AppData\Local\Programs\Python\Python36\Scripts>pip uninstall instaLooter
Uninstalling instaLooter-0.14.0:
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter-0.14.0.dist-info\description.rst
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter-0.14.0.dist-info\entry_points.txt
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter-0.14.0.dist-info\installer
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter-0.14.0.dist-info\metadata
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter-0.14.0.dist-info\metadata.json
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter-0.14.0.dist-info\record
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter-0.14.0.dist-info\top_level.txt
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter-0.14.0.dist-info\wheel
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter-0.14.0.dist-info\zip-safe
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__init__.py
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__main__.py
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__pycache__\__init__.cpython-36.pyc
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__pycache__\__main__.cpython-36.pyc
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__pycache__\batch.cpython-36.pyc
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__pycache__\cli.cpython-36.pyc
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__pycache__\core.cpython-36.pyc
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__pycache__\urlgen.cpython-36.pyc
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__pycache__\utils.cpython-36.pyc
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__pycache__\worker.cpython-36.pyc
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\batch.py
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\cli.py
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\core.py
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\urlgen.py
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\utils.py
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\worker.py
  c:\users\rajnish.rana\appdata\local\programs\python\python36\scripts\instalooter.exe
Proceed (y/n)? y
  Successfully uninstalled instaLooter-0.14.0

**2. pip install instaLooter

Output:**

C:\Users\rajnish.rana\AppData\Local\Programs\Python\Python36\Scripts>pip install instaLooter
Collecting instaLooter
  Using cached instaLooter-0.14.0-py2.py3-none-any.whl
Requirement already satisfied: python-dateutil~=2.1 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages (from instaLooter)
Requirement already satisfied: beautifulsoup4~=4.4 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages (from instaLooter)
Requirement already satisfied: progressbar2~=3.1 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages (from instaLooter)
Requirement already satisfied: hues<1,>=0.2.2 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages (from instaLooter)
Requirement already satisfied: requests~=2.18 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\requests-2.18.4-py3.6.egg (from instaLooter)
Requirement already satisfied: six~=1.4 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages (from instaLooter)
Requirement already satisfied: docopt~=0.4 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages (from instaLooter)
Requirement already satisfied: python-utils>=2.3.0 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages (from progressbar2~=3.1->instaLooter)
Requirement already satisfied: PyYAML in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages (from hues<1,>=0.2.2->instaLooter)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\chardet-3.0.4-py3.6.egg (from requests~=2.18->instaLooter)
Requirement already satisfied: idna<2.7,>=2.5 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\idna-2.6-py3.6.egg (from requests~=2.18->instaLooter)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\urllib3-1.22-py3.6.egg (from requests~=2.18->instaLooter)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\certifi-2018.1.18-py3.6.egg (from requests~=2.18->instaLooter)
Installing collected packages: instaLooter
Successfully installed instaLooter-0.14.0

**3. pip install --user -U https://github.com/althonos/InstaLooter/archive/dev.zip

Output:**

C:\Users\rajnish.rana\AppData\Local\Programs\Python\Python36\Scripts>pip install --user -U https://github.com/althonos/InstaLooter/archive/dev.zip
Collecting https://github.com/althonos/InstaLooter/archive/dev.zip
  Downloading https://github.com/althonos/InstaLooter/archive/dev.zip (71kB)
    100% |████████████████████████████████| 71kB 174kB/s
Installing collected packages: UNKNOWN
  Running setup.py install for UNKNOWN ... done
Successfully installed UNKNOWN-0.0.0

**4. opened cmd in windows system and ran this command: instaLooter inspirational.theory

Output:**

C:\Users\rajnish.rana\AppData\Local\Programs\Python\Python36\Scripts>instaLooter inspirational.theory
15:37:12 - WARNING - Could not find page of user: inspirational.theory
15:37:12 - WARNING - No medias found.

I don't know what I am doing wrong. Can you please tell me which step I need to rectify?

@lynxrv21

This comment has been minimized.

Copy link
Author

commented Mar 21, 2018

@rajnishcseku try to skip step 2

@djvanguard

This comment has been minimized.

Copy link

commented Mar 21, 2018

I did similar to @rajnishcseku but I did skip step 2. (So just uninstall, and then the install of the dev.zip)
It also says it installed UNKNOWN-0.0.0 and then doesn't seem to know what InstaLooter is when I try to run it. Do I need to find and extract the dev.zip somewhere?

@JonasGrobe

This comment has been minimized.

Copy link

commented Mar 21, 2018

@djvanguard pip install setuptools --upgrade, then try again

@rajnishcseku

This comment has been minimized.

Copy link

commented Mar 21, 2018

@JonasGrobe I ran the command given by you also uninstalled the instaLooter (i.e. skipping step 2)

Now when I try to run command "instaLooter inspirational.theory" output comes:

c:>instaLooter inspirational.theory
'instaLooter' is not recognized as an internal or external command,
operable program or batch file.

Do I need to set some some kind of environment variable in my laptop or is there any other problem?

@althonos

This comment has been minimized.

Copy link
Owner

commented Mar 21, 2018

@rajnishcseku: and on python -m instalooter inspirational.theory ?

@oferchen

This comment has been minimized.

Copy link

commented Mar 21, 2018

@althonos
it took some effort I added some entries to init,
try:
from .cli import main
from .core import InstaLooter
from . import urlgen
except ImportError:
pass

while getting a bit disappointing result,
invalid format for --time parameter: None
(format is [D]:[D] where D is an ISO 8601 date)

I never user --time-parameter in original command line and also login doesn't work

@rajnishcseku

This comment has been minimized.

Copy link

commented Mar 22, 2018

@althonos I ran the command python -m instalooter inspirational.theory
python -m instaLooter inspirational.theory

Output:

C:\Users\rajnish.rana>python -m instalooter inspirational.theory
C:\Users\rajnish.rana\AppData\Local\Programs\Python\Python36\python.exe: No module named instaLooter

C:\Users\rajnish.rana>python -m instaLooter inspirational.theory
C:\Users\rajnish.rana\AppData\Local\Programs\Python\Python36\python.exe: No module named instaLooter

@oferchen

This comment has been minimized.

Copy link

commented Mar 23, 2018

@rajnishcseku it looks like the module directory on windows is not instaLooter
but windows is not usually case sensitive so I'm not sure how python modules are
I would suggest to just rename the directory of the module from instalooter to instaLooter

@althonos

This comment has been minimized.

Copy link
Owner

commented Mar 25, 2018

An alpha release of v1.0.0 is out. I consider this issue closed, as the new structure is handled fine.

#150 was opened to fix the non-working login procedure.

@althonos althonos closed this Mar 25, 2018

@althonos althonos added the fixed label Apr 3, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
10 participants
You can’t perform that action at this time.