Instagram changed its inner data structure

[lynxrv21]

Getting the same error for any username I try.

instaLooter instagram
00:41:53 - WARNING - Could not find page of user: instagram
00:41:53 - WARNING - No medias found.

[bilyboy785]

+1

[bilyboy785]

Do you think fix this or it is normal ?

[jnskw]

Instagram might have changed the structure of its data.

I think Instagram-Scraper had a similar Problem:
https://github.com/rarcega/instagram-scraper/commit/3840aab1386f5923aa34827b688d67944a8a4ae2

[lynxrv21]

So, is anyone going to fix it here, or I have to do it by myself? :)

[bilyboy785]

if you fix it, can you share it ? :)

[lynxrv21]

No luck with this yet.
Looks like we need @althonos to help us.

[bilyboy785]

Only he does not look active anymore.

[althonos]

My studies are calling, so I don't have much time to invest in this.
The main issue is that IG changed not only its internal data representation, but also the method used to query those data. This is what makes fixing a pain. I'll try to see what I can do.

[AbysmalBiscuit]

Hey,

I've had a look at the new json structure, and as @althonos says it's different from what instaLooter expects to find, hence why there is an internal KeyError when the response data gets processed for the firt time in core.py by:
media_info = data['entry_data'][self._page_name][0][self._section_name]['media']

From analyzing the json reponse, all the information that may be wanted is in there: image src url, metadata etc...

This is type of code that's needed to get what I think should be all the media info that instaLooter expects ( this will just print the response contents of all posts loaded in the first response):
https://gist.github.com/AbysmalBiscuit/0799d5ed77e9ec6415836d8c8da9fd2d

[althonos]

I've got a working-ish fix on the dev branch you can install with:

pip install --user --upgrade https://github.com/althonos/InstaLooter/archive/6d6dac1.zip

I still want to check the code in greater detail before pushing a release, but at least it downloads pics as intended for both users and hashtags.

[AbysmalBiscuit]

@althonos Thanks for the quick fix!

Edit: Lines 339 and 340 have a pprint call:
from pprint import pprint
pprint(data['data'][self._section_name].keys())
that prints dict_keys(['edge_owner_to_timeline_media']) with every page that gets loaded, but otherwise everything seems to work great. :)

[althonos]

@AbysmalBiscuit : fixed, thanks.

[oferchen]

another issue could be related,
Exception in thread Thread-5:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/local/lib/python2.7/dist-packages/instaLooter/worker.py", line 70, in run
self._download_photo(media)
File "/usr/local/lib/python2.7/dist-packages/instaLooter/worker.py", line 133, in _download_photo
self._add_metadata(photo_name, media)
File "/usr/local/lib/python2.7/dist-packages/instaLooter/worker.py", line 106, in _add_metadata
self._get_caption(metadata).encode('utf-8'),
File "/usr/local/lib/python2.7/dist-packages/instaLooter/worker.py", line 45, in _get_caption
return metadata['edge_media_to_caption']['edges'][0]['node']['text']
IndexError: list index out of range

[ghost]

Using timeframes still doesn't work - had to change line 416 in core.py from "for media in page[media]['edges']:" to "for media in page[medias]['edges']:"

[Bigben83]

I pulled the dev branch and the could not find page of user is still showing ? should i be doing something else ?

[althonos]

@Bigben83 : could you give me some parameters so that I can reproduce this ? It works just fine with a public user on my machine.

[Bigben83]

@althonos this is what i have been using
/usr/local/bin/instaLooter sawyerhartman /home/dev/public_html/instagram/sawyerhartman -T {username}.{datetime} -q --add-metadata -v -N -n 50

[rajnishcseku]

@althonos I am also having the same issue which @Bigben83 is having.

[althonos]

@Bigben83 @rajnishcseku @JonasGrobe @oferchen @AbysmalBiscuit @jnskw @lynxrv21 @bilyboy785 : If you want to test the new version, you can pull from the dev branch.

The codebase is brand new, but the CLI should be very familiar, and most problems should be fixed. Note that I didn't test the login at all since I have no IG account and I don't know if the login procedure used before still works !

I'll release to PyPI once I've got the documentation updated and the API stabilised.

In the meantime:

pip install --user -U https://github.com/althonos/InstaLooter/archive/dev.zip

Oh, and by the way, you cannot directly download a profile without using the instalooter user subcommand. This is easier for both docopt and myself 😉

[rajnishcseku]

@althonos

Hi, I followed following steps:

1. pip uninstall instaLooter

Output:

C:\Users\rajnish.rana\AppData\Local\Programs\Python\Python36\Scripts>pip uninstall instaLooter
Uninstalling instaLooter-0.14.0:
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter-0.14.0.dist-info\description.rst
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter-0.14.0.dist-info\entry_points.txt
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter-0.14.0.dist-info\installer
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter-0.14.0.dist-info\metadata
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter-0.14.0.dist-info\metadata.json
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter-0.14.0.dist-info\record
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter-0.14.0.dist-info\top_level.txt
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter-0.14.0.dist-info\wheel
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter-0.14.0.dist-info\zip-safe
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__init__.py
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__main__.py
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__pycache__\__init__.cpython-36.pyc
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__pycache__\__main__.cpython-36.pyc
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__pycache__\batch.cpython-36.pyc
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__pycache__\cli.cpython-36.pyc
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__pycache__\core.cpython-36.pyc
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__pycache__\urlgen.cpython-36.pyc
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__pycache__\utils.cpython-36.pyc
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\__pycache__\worker.cpython-36.pyc
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\batch.py
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\cli.py
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\core.py
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\urlgen.py
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\utils.py
  c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\instalooter\worker.py
  c:\users\rajnish.rana\appdata\local\programs\python\python36\scripts\instalooter.exe
Proceed (y/n)? y
  Successfully uninstalled instaLooter-0.14.0

**2. pip install instaLooter

Output:**

C:\Users\rajnish.rana\AppData\Local\Programs\Python\Python36\Scripts>pip install instaLooter
Collecting instaLooter
  Using cached instaLooter-0.14.0-py2.py3-none-any.whl
Requirement already satisfied: python-dateutil~=2.1 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages (from instaLooter)
Requirement already satisfied: beautifulsoup4~=4.4 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages (from instaLooter)
Requirement already satisfied: progressbar2~=3.1 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages (from instaLooter)
Requirement already satisfied: hues<1,>=0.2.2 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages (from instaLooter)
Requirement already satisfied: requests~=2.18 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\requests-2.18.4-py3.6.egg (from instaLooter)
Requirement already satisfied: six~=1.4 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages (from instaLooter)
Requirement already satisfied: docopt~=0.4 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages (from instaLooter)
Requirement already satisfied: python-utils>=2.3.0 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages (from progressbar2~=3.1->instaLooter)
Requirement already satisfied: PyYAML in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages (from hues<1,>=0.2.2->instaLooter)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\chardet-3.0.4-py3.6.egg (from requests~=2.18->instaLooter)
Requirement already satisfied: idna<2.7,>=2.5 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\idna-2.6-py3.6.egg (from requests~=2.18->instaLooter)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\urllib3-1.22-py3.6.egg (from requests~=2.18->instaLooter)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\rajnish.rana\appdata\local\programs\python\python36\lib\site-packages\certifi-2018.1.18-py3.6.egg (from requests~=2.18->instaLooter)
Installing collected packages: instaLooter
Successfully installed instaLooter-0.14.0

**3. pip install --user -U https://github.com/althonos/InstaLooter/archive/dev.zip

Output:**

C:\Users\rajnish.rana\AppData\Local\Programs\Python\Python36\Scripts>pip install --user -U https://github.com/althonos/InstaLooter/archive/dev.zip
Collecting https://github.com/althonos/InstaLooter/archive/dev.zip
  Downloading https://github.com/althonos/InstaLooter/archive/dev.zip (71kB)
    100% |████████████████████████████████| 71kB 174kB/s
Installing collected packages: UNKNOWN
  Running setup.py install for UNKNOWN ... done
Successfully installed UNKNOWN-0.0.0

**4. opened cmd in windows system and ran this command: instaLooter inspirational.theory

Output:**

C:\Users\rajnish.rana\AppData\Local\Programs\Python\Python36\Scripts>instaLooter inspirational.theory
15:37:12 - WARNING - Could not find page of user: inspirational.theory
15:37:12 - WARNING - No medias found.

I don't know what I am doing wrong. Can you please tell me which step I need to rectify?

[lynxrv21]

@rajnishcseku try to skip step 2

[djvanguard]

I did similar to @rajnishcseku but I did skip step 2. (So just uninstall, and then the install of the dev.zip)
It also says it installed UNKNOWN-0.0.0 and then doesn't seem to know what InstaLooter is when I try to run it. Do I need to find and extract the dev.zip somewhere?

[ghost]

@djvanguard pip install setuptools --upgrade, then try again

[rajnishcseku]

@JonasGrobe I ran the command given by you also uninstalled the instaLooter (i.e. skipping step 2)

Now when I try to run command "instaLooter inspirational.theory" output comes:

c:>instaLooter inspirational.theory
'instaLooter' is not recognized as an internal or external command,
operable program or batch file.

Do I need to set some some kind of environment variable in my laptop or is there any other problem?

[althonos]

@rajnishcseku: and on python -m instalooter inspirational.theory ?

[oferchen]

@althonos
it took some effort I added some entries to init,
try:
from .cli import main
from .core import InstaLooter
from . import urlgen
except ImportError:
pass

while getting a bit disappointing result,
invalid format for --time parameter: None
(format is [D]:[D] where D is an ISO 8601 date)

I never user --time-parameter in original command line and also login doesn't work

[rajnishcseku]

@althonos I ran the command python -m instalooter inspirational.theory
python -m instaLooter inspirational.theory

Output:

C:\Users\rajnish.rana>python -m instalooter inspirational.theory
C:\Users\rajnish.rana\AppData\Local\Programs\Python\Python36\python.exe: No module named instaLooter

C:\Users\rajnish.rana>python -m instaLooter inspirational.theory
C:\Users\rajnish.rana\AppData\Local\Programs\Python\Python36\python.exe: No module named instaLooter

[oferchen]

@rajnishcseku it looks like the module directory on windows is not instaLooter
but windows is not usually case sensitive so I'm not sure how python modules are
I would suggest to just rename the directory of the module from instalooter to instaLooter

[althonos]

An alpha release of v1.0.0 is out. I consider this issue closed, as the new structure is handled fine.

#150 was opened to fix the non-working login procedure.