Adding additional functionality of the Kijiji API

[SomeMashedPotatoes]

Firstly I'd like to say great work on this project it is very useful and well done. However, I have a particular use case that requires additional actions like querying ad searches or sending messages etc and I wanted to see how I can extend to have this functionality. I came across #13 and I have tried to carry out a MITM to inspect the network traffic from an android emulator by following this guide https://multigesture.net/articles/inspecting-https-network-traffic-of-any-android-app/ however I was unable to run the kijiji app and I instead got a message saying that the app needs google play services to run and my device does not support it, despite pushing relevant apks onto my emulator.

So I was hoping you could kindly share your experience on how you went about inspecting the network traffic from the kijiji app or what tutorials/steps did you follow.

Thanks

[rybodiddly]

I'm actually working on those exact functionalities. So with a little patience, you'll start to see them integrated into the project.

You might want to use an actual device opposed to a vm. The kijiji app exists for both iphone and android devices. So lots of options. Most mitm apps supply you with a certificate to add to the device. Mitmproxy should work fine. Haven't used it, but principles are the same. I also assume you will need to enable ssl proxying in the mitm app.

[SomeMashedPotatoes]

Oh that's awesome. That's great advice yeah it was much easier to just set up a proxy for outgoing network requests from my iphone to my computer. I took a look at the endpoints for like sending messages and searching with filters and it looks relatively straightforward.

Although this is my first time viewing the network traffic of a mobile api and I am just wondering whether Kijiji has any system in place to restrict api calls from clients other than their app? Looking at this repository and others like it, it does not seem they care enough to have done anything, so It makes me think they are probably okay with us accessing the internal api like this? What do you think? Although I assume they may have some sort of request throttling to avoid DDoS attacks and the like but that's standard for any public website/api.

[SomeMashedPotatoes]

Actually yeah Kijiji mentions in their terms of service to not do any sort of scraping or automation. So i was wondering if you encountered any sort of rate limiting and automation detection for the API.

[rybodiddly]

Best advice is simply don't abuse / spam the mingle server. The server will deny access to abusive accounts or place restrictions on flagged accounts. They use a threat matrix to determine problematic accounts.

In terms of additional features,the messaging / conversations api already exists within the api file and can be modified to suit your needs. Search functionality is almost complete. Will update the api file in a few days with a search function. Will take a little more time to incorporate it into the web gui.

[SomeMashedPotatoes]

That makes sense although "spam" can be difficult to quantify. For example, I wanted to create something that will search for new ads as much as frequently as possible and then potentially send messages to the ad poster. I was just wondering from what you've tested what kind of rate limiting they do. Like would doing the search every minute and sending messages to every new ad since the last query be filtered as spam? Also would taking secondary actions like sending messages actually hinder spam detection since usually systems like these take into account other actions that were taken beyond just querying/scraping content. Then there's also the consideration of using different web proxies to avoid being blacklisted or put on a watch list. I'm just trying to gauge whether doing something like this will end up being worthwhile since for this system I am trying to make, getting quick and consistent access to the newest ads is the most important part. I guess I should play around with it and try and find the sweet spot where I am still querying the ads pretty often but not often enough that the behaviour looks suspicious.

That's great will take a look again. Thanks for the work and the advice.

[rybodiddly]

Spam Definition

Generally spam is the bulk mailing of unwanted or unsolicited messages. So I have to wonder what you could possibly be wanting to send out at such a high intensity, as your desired modifications fit that exact definition. The fact that you are concerned with what types of spam detection kijiji has in place and if ip masking techniques are viable, should really answer your own questions. Personally, it sounds reminiscent of previous automated kijiji scams designed to search for specific high ticket items and send out automated inquiries, which scanned for specific responses and used customized reply routines that eventually convince the user to mail the item out of jurisdiction. It should be stated that I do not condone such practices in any way whatsoever. The simplest way to look at it, would be to think about what type of behaviours would be possible on your mobile app and attempt to replicate them.

The mingle server is in no way robust. Most resources seem geared towards the web interface. During initial testing, the mingle server was flaky and bogged down regularly. The threat detection metrics will not only blacklist ip's, but will block accounts as well. So even with the use of vpn's etc., accounts determined to be overpopulating the server will be shutdown. Additionally, more common techniques deployed are to not actually block the account, but to delete new listings immediately after being posted, so as to not raise suspicions and cause confusion. Threat metrics looks for specific patternings. The way to avoid it is with the incorporation of randomness. So searching or replying at regularly high intensity intervals such as 1 minute is guaranteed to be detected sooner or later.

Hope that helps clarify your concerns.