Thread Per User #2956
Thread Per User #2956
Conversation
- BREAKING: interactive password prompt removed - Removes "thread_delay" as it doesn't really serve a purpose - Removed "num_threads" since it's now directly tied to number of accounts - `-u/-p/-a` parsing is handled the same, but with better error messages and better interal list handling - Removed lat/lon global state from `config` - Fixed error where turning off scanning for long periods of time would fail to restart the scanner - Removed a bit of dead code - search_overseer now creates threads as needed - "next location" is no longer hanlded through global config - "next location" is handled more gracefully (search_threads no longer aware of it)
|
I'd love to give this a test I'm currently using a different one but it has a few hiccups. EDIT: also what syntax are you using for the multiple accounts? |
Then rebuild and run it. |
|
Not sure what the problem is but just like the latest develop build as soon as I execute runserver it closes. |
|
It should give you some output. Add |
|
I guess the one click install is broken right now. |
|
@chuyskywalker This is great stuff. I'll set it up now and let it run overnight. |
|
If we're now assigning a user per thread, is there any way to mark this in the database so that multiple workers can be passed the same set of credentials and they won't reuse them? |
|
I'm using a very large number of accounts -st 6 with 3 on each instance. |
|
Obviously I am doing something wrong? Defined my Users in the config.ini: Started with the following: |
* Add support for multiple accounts (#2546) Changes program arguments: -u/--user and -p/--password have been removed. -a/--auth has been changed to -a/--auths and takes in one or more strings of form: auth_provider:username:password auth_provider is ptc or google. If there are n accounts, search thread i will use account (i % n). * Fix moving credentials to lowercase * Add backwards compatibility to multiple accounts Multiple accounts are no longer given as strings containing auth service, username and password, because that was not backwards compatible. Now -a, -u and -p each can be given multiple times. With this patch multiple usernames can be given. They either use one password for all accounts or unique password for each account can be passed. Same goes for auth services. Auth defaults to ptc. Unfortunately the way argparse handles config files each these cannot take multiple parameters. '-u user1 user2' is not possible, needs to be '-u user1 -u user2' The following parameters should be ok: Using single account should be just as it was earlier: ./runserver.py -a ptc -u user -p pass -st 3 -t 3 ... This will use ptc for all accounts and ask interactively for one password to use for all accounts: ./runserver.py -u user1 -u user2 -u user3 -st 3 -t 3 ... This will use password 'pass' for all accounts and use ptc: ./runserver.py -u user1 -u user2 -u user3 -p pass -st 3 -t 3 ... This will use google for all accounts and unique passwords for each account: ./runserver.py -a google -u user1 -u user2 -p pass1 -p pass2 ... This will log one in ptc and one in google with respective un/pw: ./runserver.py -a ptc -a google -u user1 -u user2 -p pass1 -p pass2 ... In config.ini these can be given as lists or single values: auth-service: [google, ptc] username: [user1, user2] password: onepasswordforbothaccounts * Add support for multiple accounts (#2546) Changes program arguments: -u/--user and -p/--password have been removed. -a/--auth has been changed to -a/--auths and takes in one or more strings of form: auth_provider:username:password auth_provider is ptc or google. If there are n accounts, search thread i will use account (i % n). * Fix moving credentials to lowercase * Add backwards compatibility to multiple accounts Multiple accounts are no longer given as strings containing auth service, username and password, because that was not backwards compatible. Now -a, -u and -p each can be given multiple times. With this patch multiple usernames can be given. They either use one password for all accounts or unique password for each account can be passed. Same goes for auth services. Auth defaults to ptc. Unfortunately the way argparse handles config files each these cannot take multiple parameters. '-u user1 user2' is not possible, needs to be '-u user1 -u user2' The following parameters should be ok: Using single account should be just as it was earlier: ./runserver.py -a ptc -u user -p pass -st 3 -t 3 ... This will use ptc for all accounts and ask interactively for one password to use for all accounts: ./runserver.py -u user1 -u user2 -u user3 -st 3 -t 3 ... This will use password 'pass' for all accounts and use ptc: ./runserver.py -u user1 -u user2 -u user3 -p pass -st 3 -t 3 ... This will use google for all accounts and unique passwords for each account: ./runserver.py -a google -u user1 -u user2 -p pass1 -p pass2 ... This will log one in ptc and one in google with respective un/pw: ./runserver.py -a ptc -a google -u user1 -u user2 -p pass1 -p pass2 ... In config.ini these can be given as lists or single values: auth-service: [google, ptc] username: [user1, user2] password: onepasswordforbothaccounts * Merge fix
|
I went from 10minutes logging in to 3 seconds logging in. Big thanks, enormous improvement, for the first time I feel like the old maps are actually back. I hope this gets merged soon! |
|
i need this in my life |
|
How are multiple accounts defined in |
I don't know why that would happen. It would seem python/environment specific, since that's an internal function which is erroring out. See if google leads you down any paths. It's config supported the same as the current develop branch -- didn't change how it works, Frozen left an example a few comments up. @archalias -- thanks for testing; I'm glad to see it's just working smoother. This patch won't make the looping go (generally) any faster, as you've seen. It does fix the "everyone has to login first" startup problem though. |
|
@chuyskywalker Yes I observed that this error only gets thrown when I have an high amount of Accounts (200+) I guess my Security settings preventing the additional Thread. |
|
The other PR got merged, so the file diff on this one now makes sense, if any one wishes to review it with clarity |
|
but with this you can't set multiple workers because the other worker uses also the config.ini with the same accounts! |
|
You can use different accounts per script invocation. |
|
This isn't the right thread to look for support, please. |
|
Checked this out, running with (only?) 10 accounts at the moment, but holy hell this is fast and super accurate (as far as I can test this). Great stuff. Good job. |
|
When I start scanner with 50+ accounts (1 string), 10% of accounts can't login. Maybe add some delay between login period? But the other things - working like a charm! |
|
Then view it as not asking for advice but as putting to light an existing issue. It's an immediate result of the many threads requiring write access. |
|
+1, just need to update the documentation and config.example.ini files! |
|
@invisiblek Let's get this +1 and merged! |
|
@chuyskywalker I know that this PR is already merged, but I have a question regarding the change: Is this PR adding the threads dynamically based on the accounts that were able to log in or at startup time based on the number of provided accounts? It would be awesome if it scans with all google accounts, even if the PTC login is currently unavailable 😉 |
|
There is a bug - current_location for Flask is not updating after location change, so after page refresh, marker will return to origin. I've made simple fix in #2980 |
|
This is not suitable for windows, is it? What am I doing wrong? Edit: If I first start the normal developement branch and afterwards the customized (by this thread) all pokemon will be visable on one map(of both versions). |
|
I'm getting a "could not decode JSON" while trying to log in ... |
I believe what should happen is that the, let's say, PTC account thread(s) will be stuck in a loop trying to login constantly. The Google account thread(s) would continue to peel search items off the list. However, some of the items would be stuck in the PTC thread(s) so the queue would never reach zero. Thus the overseer could never "refill" the queue with the search areas again. Adding a "max tries before we give it back" option would be a good idea if there is more than one search worker thread. That'd need to come in another PR down the road though. |
|
@hodoliiiii run |
Description
I have no doubt that we're never getting multi-threading per account back. We can, however, do a better job at threading across multiple accounts.
This does that, and cleans up some other things related.
Motivation and Context
Wanted to fix a bug wherein the searcher pause would not return after a while. Realized I need to move the
PgoAPIwaaay deeper into the system to make that work. Things snowballed a little bit, but I made a lot of, what I consider, to be good improvements.-u/-p/-aparsing is handled the same, but with better error messages and better intenral list handlingconfigHow Has This Been Tested?
Not very extensively yet. In fact, I accidentally (and sorta stupidly) built it from #2863, so it looks like a lot more code than it really is. That other PR should get merged first; then this one will look less all over the place.
That all said, this needs a lot more attention for all the various modes and methods we support. (Unit/Integration testing? What's that!?)
Types of changes
Checklist:
(Doc updates will definitely be in order with this one -- but there, generally speaking, seems to be a huge need for "HEY, SCANNING WORKS LIKE XYZ NOW, THINGS CHANGED" type announcement.