Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop collection #21

Closed
SiIason opened this issue Jan 27, 2021 · 57 comments · Fixed by #26
Closed

Drop collection #21

SiIason opened this issue Jan 27, 2021 · 57 comments · Fixed by #26
Labels
enhancement New feature or request

Comments

@SiIason
Copy link

SiIason commented Jan 27, 2021

Current repo implementation does not count viewing time towards timed Drop rewards.

The example I looked at needs to be claimed (button clicked) every hour to continue earning towards watched time. So simply going afk only ever builds up time for 1 drop. You can Claim via a popup message box in chat (same as channel points) or via the inventory page, you could have multiple claim buttons here. I've included code for both buttons should they require css descriptors.

twitchchatclaimbutton.txt
twitchinventoryclaimbutton.txt

I assume this also means other types of Drop reward are not able to be collected?

Solution could be to keep the channel stream open, which I don't believe currently happens, clicking the claim button when it appears. Most pages have the "drops enabled" tag so wouldn't need to happen if no drops enabled. Or a periodic check of your inventory for available items to be claimed could be a simpler catch-all (assuming watch progress starts to be recorded).

@SiIason SiIason added the enhancement New feature or request label Jan 27, 2021
@Tkd-Alex
Copy link
Owner

In Italy the use of all Twitch functionality is not diffused, and also I am not an expert.

Can you provide also a screenshot for this drop reward/button etc? Maybe this can help me understand ...

Anyway, I want to reduce the usage of Selenium, currently is required only for the Bet, and I've also written a POC for do only with API. We should implement also this 'Drop collection function' with API if It's possible

@SiIason
Copy link
Author

SiIason commented Jan 27, 2021

Inventory During: https://prnt.sc/xq5n15
Inventory Ready: https://prnt.sc/xq5me4

@Tkd-Alex
Copy link
Owner

Tkd-Alex commented Jan 27, 2021

Sorry bro but I don't know what are you talking about 😂

Any steps to reproduce so I can try by myself? Maybe it's related to the watch increasing-time in based on chat, like the issue at this link

gottagofaster236/Twitch-Channel-Points-Miner#16

@Tkd-Alex
Copy link
Owner

Also I don't know If we have a different endpoints or xpath buttons for each games. I think the drops are game related, right?

@Rakambda
Copy link
Contributor

To me it looks similar to https://www.twitch.tv/drops/campaigns

@SiIason
Copy link
Author

SiIason commented Jan 28, 2021

Yes this is partially the same issue.

  • The first part would need to have watch time increase as outlined within your linked discussion.
  • The second part would be to check for available claim collection, as each need to be claimed/clicked before your watch time starts counting to the next Drop.

The screenshots I posted as from "https://www.twitch.tv/drops/inventory" showing one from a collection of drops partially rewarded and then the second image shows it finished where the claim button appears. This needs clicking for the next drop to start unlocking, its not automatic.

@Tkd-Alex
Copy link
Owner

OK, so for testing I should link my Twitch account on one of games with drops enable.
The problem:

  • I'm on Linux
  • I'm not a Gamer

@SiIason
Copy link
Author

SiIason commented Jan 28, 2021

https://www.protondb.com/app/1049590

But happy to test :)

@Tkd-Alex
Copy link
Owner

Do you think It's possible to register and account on this game, linked on Twitch and claim the bonus without install or play the game? 😆 Really, I've too much problem with my PC. First I've not enough space, I should format everything and change SSD, but I want to complete my University thesis first.

@Tkd-Alex
Copy link
Owner

Tkd-Alex commented Jan 28, 2021

For increase the watching-time we could integrate one of the following projects:

If someone wants to try.

@SiIason
Copy link
Author

SiIason commented Jan 28, 2021

I tried using Chatterino today to see if joining chat increased watch time reward status but it didn't.
I'll test the above repos you've linked for results and feedback.

@Tkd-Alex
Copy link
Owner

Guys the view-count for drops It's not chat-related.
I've done just now multiple test and analysis. I think I found a way to do that.

This is a proof, every minute the value on currentMinutesWatched It's increased

image

The only update we should to do Is to add the game name in the requested with event: minute-watched.
Some references:

def update_minute_watched_event_request(self, streamer):
event_properties = {
"channel_id": streamer.channel_id,
"broadcast_id": self.get_broadcast_id(streamer),
"player": "site",
"user_id": self.twitch_login.get_user_id(),
}
minute_watched = [{"event": "minute-watched", "properties": event_properties}]
json_event = json.dumps(minute_watched, separators=(",", ":"))
streamer.minute_watched_requests = RequestInfo(
self.get_minute_watched_request_url(streamer),
{"data": (b64encode(json_event.encode("utf-8"))).decode("utf-8")},
)
def get_minute_watched_request_url(self, streamer):
headers = {"User-Agent": self.user_agent}
main_page_request = requests.get(
streamer.streamer_url, headers=headers
)
response = main_page_request.text
settings_url = re.search(
"(https://static.twitchcdn.net/config/settings.*?js)", response
).group(1)
settings_request = requests.get(
settings_url, headers=headers
)
response = settings_request.text
minute_watched_request_url = re.search('"spade_url":"(.*?)"', response).group(1)
return minute_watched_request_url

response = requests.post(
streamers[index].minute_watched_requests.url,
data=streamers[index].minute_watched_requests.payload,
headers={"User-Agent": self.user_agent},
)

event_properties = {
    "channel_id": 123456,
    "broadcast_id": 123456,
    "player": "site",
    "user_id": 123456,
    "game": "Eternal Return: Black Survival",
}
minute_watched = [{"event": "minute-watched", "properties": event_properties}]
json_event = json.dumps(minute_watched, separators=(",", ":"))
payload = {"data": (b64encode(json_event.encode("utf-8"))).decode("utf-8")}

response = requests.post('https://spade-url', headers=headers, data=payload)

This is the CURL request for check if the counter is increased:

curl 'https://gql.twitch.tv/gql' \
  -H 'Connection: keep-alive' \
  -H 'Pragma: no-cache' \
  -H 'Cache-Control: no-cache' \
  -H 'Authorization: OAuth <write-your-token-here>' \
  -H 'DNT: 1' \
  -H 'Accept-Language: it-IT' \
  -H 'Client-Id: kimne78kx3ncx6brgo4mv6wki5h1ko' \
  -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36' \
  -H 'X-Device-Id: 0433ffc66c638dde' \
  -H 'Content-Type: text/plain;charset=UTF-8' \
  -H 'Accept: */*' \
  -H 'Origin: https://www.twitch.tv' \
  -H 'Sec-Fetch-Site: same-site' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Referer: https://www.twitch.tv/drops/inventory' \
  --data-raw '{"operationName":"DropCurrentSessionContext","variables":{},"extensions":{"persistedQuery":{"version":1,"sha256Hash":"2e4b3630b91552eb05b76a94b6850eb25fe42263b7cf6d06bee6d156dd247c1c"}}}' \
  --compressed

Anyway, I think I'll do initially a new branch just for test the increase time.
After that we can work on drops collection. I've already analysed some API for manage the inventory.

@SiIason
Copy link
Author

SiIason commented Jan 28, 2021

Wow, that's fantastic work.
Can't wait to see this in practice.

Tkd-Alex added a commit that referenced this issue Jan 28, 2021
…drops. Insert the game name (if drops are enabled) in minute_watched request. Ref #21
@Tkd-Alex
Copy link
Owner

Guys, I need some testers please! Someone can try this branch on latest commit? 🙏 @SiIason ?
https://github.com/Tkd-Alex/Twitch-Channel-Points-Miner-v2/tree/drops-watchtime

I need also If it's possible a collection of all message starting with: Drop update

28/01 23:45:04 - Drop update: {'type': 'drop-progress', 'data': {'drop_id': 'xxxx-yyyy-zzzz', 'channel_id': '123456', 'current_progress_min': 24, 'required_progress_min': 60}}

@SiIason
Copy link
Author

SiIason commented Jan 29, 2021

Working great... Checked with Twitch and the reward counter is increasing.

29/01/21 01:20:49 - INFO - [on_message]: Drop update: {'type': 'drop-progress', 'data': {'drop_id': 'cfd9a6de-13af-44dc-a85a-8e78695d9483', 'channel_id': '77314997', 'current_progress_min': 27, 'required_progress_min': 60}}
29/01/21 01:21:51 - INFO - [on_message]: Drop update: {'type': 'drop-progress', 'data': {'drop_id': 'cfd9a6de-13af-44dc-a85a-8e78695d9483', 'channel_id': '77314997', 'current_progress_min': 28, 'required_progress_min': 60}}

@Tkd-Alex
Copy link
Owner

Working great... Checked with Twitch and the reward counter is increasing.

29/01/21 01:20:49 - INFO - [on_message]: Drop update: {'type': 'drop-progress', 'data': {'drop_id': 'cfd9a6de-13af-44dc-a85a-8e78695d9483', 'channel_id': '77314997', 'current_progress_min': 27, 'required_progress_min': 60}}
29/01/21 01:21:51 - INFO - [on_message]: Drop update: {'type': 'drop-progress', 'data': {'drop_id': 'cfd9a6de-13af-44dc-a85a-8e78695d9483', 'channel_id': '77314997', 'current_progress_min': 28, 'required_progress_min': 60}}

Thanks for your time.
Have you other logs? Because currently we have only the type: "drop-progress".

And I would other type for continue the implementation of this. For example I could check every update if current_progress_min is equal to required_progress_min but I'd like that Twitch give use other messages type, something like: drop-complete and/or drop-claimed

@Rakambda
Copy link
Contributor

Seems like it works for some drops but not all?

Like for now on Rainbow Six streams, drops are activated (eg: https://www.twitch.tv/kinggeorge).

This line is fireing well and I see the game in the logs
a72a327#diff-72bedccfed7f2e53b7585026f3c3ba3daf715d0f7acd3504553cb281bb649c61R62

Though the drops are not detected as this line doesn't fire
a72a327#diff-72bedccfed7f2e53b7585026f3c3ba3daf715d0f7acd3504553cb281bb649c61R67

Now maybe a difference is that these drops doesn't appear in the page https://www.twitch.tv/drops/campaigns .

@Tkd-Alex
Copy link
Owner

For the moment I've enabled the increase counters only for the streams with "Drop enable" tag.

Screenshot_20210129-094122__01.jpg

Missing here. I seen that the drops are enabled only in the title and in the chat, with stream elements and command !drops

I don't know if this drops type are developed by third part, and not directly form Twitch 🤔 This is only an Idea but It's possibile that If this drop are developed not from Twitch the counter Is chat based, test with chatterino if the counter increase. About the counter I'm not sure that you have a counter, because the drops event Is not in Inventory page. I want to repeat again, maybe It's something developed by Ubisoft in this case.

Test chatterino, If you notice some differences report here and we can try to find a solution. For sure we can't claim the drop for you.

@Rakambda
Copy link
Contributor

They are drops from Ubisoft, and probably require the Ubisoft account to be linked to Twitch in order to show up the drops.

image

I'll try connecting through IRC to the chat and see if it works that way.

@Tkd-Alex
Copy link
Owner

Oh! wait 😢 @RakSrinaNa
I was from mobile device and I had not read really well your comment.

You can try also to edit the following line and remove the If that search for id: c2542d6d-cd10-4532-919b-3d19f30a768b

if "c2542d6d-cd10-4532-919b-3d19f30a768b" in [tag['id'] for tag in streamer_info['stream']['tags']]:
logger.info(f"{streamer} - Drops are enabled for this stream! ")
event_properties["game"] = streamer_info['broadcastSettings']['game']['name']

if 1==1 or "c2542d6d-cd10-4532-919b-3d19f30a768b" in [tag['id'] for tag in streamer_info['stream']['tags']]:

@Tkd-Alex
Copy link
Owner

Tkd-Alex commented Jan 29, 2021

For understand as well who is chat-related and not you can try the pop up mode for player and for chat.
Leave open for x minutes of the chat and see if something change, do the same test for the player

@Rakambda
Copy link
Contributor

For now I'm running this script while sending the game everytime (with 1==1) to see if it does anything. But this will be quite hard to tell as there's no "progress bar" or anything. You only know when you get the drop itself. Probably that Ubisoft gets a list of watched time or something and decides by themselves when giving the drop?
Often seems like the processing of the watch times are done at night, so I'll know tomorrow.

I'll let it on through the day and try not to add too much time to it through Twitch.

Though looking a bit through Twitch's requests, I can see this response passing by:

{
  "data": {
    "user": {
      "id": "194258396",
      "login": "fastanne",
      "displayName": "FastAnne",
      "lastBroadcast": {
        "id": "40909281340",
        "game": {
          "id": "460630",
          "self": {
            "isDropsLinked": true,
            "__typename": "GameSelfConnection"
          },
          "activeDropCampaigns": [
            {
              "id": "fff3dad0-0efe-4048-ae4e-02ae33e79678",
              "connectionURL": "https://drops-register.ubi.com/",
              "applicableChannels": [],
              "isAvailableToAllChannels": true,
              "__typename": "DropCampaign"
            }
          ],
          "__typename": "Game"
        },
        "__typename": "Broadcast"
      },
      "stream": {
        "id": "40909281340",
        "isStreamDropsEnabled": true,
        "type": "live",
        "__typename": "Stream"
      },
      "__typename": "User"
    }
  },
  "extensions": {
    "durationMilliseconds": 59,
    "operationName": "Drops_ChannelDrops_User",
    "requestID": "01EX6VGJ6R9R87M1TDS2QJPENC"
  }
}

data.user.stream.isStreamDropsEnabled gives the indication that there's drops to be obtained.
If sending the game works maybe this value can be used to know when to send the game info.

Request:

{
  "operationName": "Drops_ChannelDrops_User",
  "variables": {
    "login": "fastanne",
    "isLoggedIn": true
  },
  "extensions": {
    "persistedQuery": {
      "version": 1,
      "sha256Hash": "f309b1d517d288074d50d96512059857cc67d8905d1379e414d70f7b981f2618"
    }
  }
}

@Tkd-Alex
Copy link
Owner

Probably that Ubisoft gets a list of watched time or something and decides by themselves when giving the drop?

Or just check who are online in chat and share randomly the drop 🤔

@Tkd-Alex
Copy link
Owner

About sending the game info we can do it every time, It's not a problem.
For the moment I've inserted the If condition base on Drop Tag only for help us to debug and reduce the data sent.

The strangest thing here If you use the same request Drops_ChannelDrops_User with one of the users founded in https://www.twitch.tv/drops/inventory all the values are different 😞

{
    "data": {
        "user": {
            "id": "00000",
            "login": "streamer",
            "displayName": "streamer",
            "lastBroadcast": {
                "id": "00000",
                "game": {
                    "id": "00000",
                    "self": {
                        "isDropsLinked": false,
                        "__typename": "GameSelfConnection"
                    },
                    "activeDropCampaigns": [],
                    "__typename": "Game"
                },
                "__typename": "Broadcast"
            },
            "stream": {
                "id": "00000",
                "isStreamDropsEnabled": false,
                "type": "live",
                "__typename": "Stream"
            },
            "__typename": "User"
        }
    },
    "extensions": {
        "durationMilliseconds": 58,
        "operationName": "Drops_ChannelDrops_User",
        "requestID": "01EX6ZWKPAGB5C6P11NPCTN3R3"
    }
}

@vadrozh
Copy link

vadrozh commented Jan 29, 2021

On R6 you doesn't needed to claim reward, it claims automatically

@Tkd-Alex
Copy link
Owner

On R6 you doesn't needed to claim reward, it claims automatically

Excuse my ignorance, R6 stand for Rainbow Six?
Do you think It's the same for all Ubisoft Drops?

@Tkd-Alex
Copy link
Owner

But It should say 'Claimed' on the claimed drop not carry on counting.

It's possible that Sector's Edge It's different from Eternal Return? I think we can claim at least 3 of this +50% Proton Boost
https://www.twitch.tv/drops/campaigns?dropID=c30f52c2-77b8-49eb-9ebb-0479590a4aa8

Also: Looks like only drops from one game are collecting.

About this I don't understand. Sorry

@SiIason
Copy link
Author

SiIason commented Jan 29, 2021

It looks stuck now. Looks like this https://prnt.sc/xtkknh

29/01/21 16:47:56 - INFO - [on_message]: Drop update: {'type': 'drop-progress', 'data': {'drop_id': '4ff2f5fc-abea-4ce6-ab5c-a55d0832ef28', 'channel_id': '174675703', 'current_progress_min': 14, 'required_progress_min': 15}}
29/01/21 16:49:17 - INFO - [on_message]: Drop update: {'type': 'drop-progress', 'data': {'drop_id': '4ff2f5fc-abea-4ce6-ab5c-a55d0832ef28', 'channel_id': '174675703', 'current_progress_min': 15, 'required_progress_min': 15}}
29/01/21 16:49:25 - INFO - [on_message]: Drop update: {'type': 'drop-progress', 'data': {'drop_id': '798c72d3-c4e8-4ac8-ad45-17737cbb35d9', 'channel_id': '522172972', 'current_progress_min': 0, 'required_progress_min': 45}}
29/01/21 16:49:55 - INFO - [on_message]: Drop update: {'type': 'drop-progress', 'data': {'drop_id': 'c18b3229-200b-4362-a561-5c7dd55c0695', 'channel_id': '174675703', 'current_progress_min': 15, 'required_progress_min': 15}}
29/01/21 16:50:57 - INFO - [on_message]: Drop update: {'type': 'drop-progress', 'data': {'drop_id': 'c18b3229-200b-4362-a561-5c7dd55c0695', 'channel_id': '174675703', 'current_progress_min': 15, 'required_progress_min': 15}}

@SiIason
Copy link
Author

SiIason commented Jan 29, 2021

About this I don't understand. Sorry

I only get "Drop update" messages from the second campaign/game when then first campaign doesn't check for drops

@SiIason
Copy link
Author

SiIason commented Jan 29, 2021

Checking again Twitch now says I've clamed two rewards (at the bottom of the page) even though the second looks unclaimed as you can see from previous screenshot and the "Drop update" log message is still stuck reporting 15/15

@Tkd-Alex
Copy link
Owner

Tkd-Alex commented Jan 29, 2021

If you can please do a git pull and test the latest version. With the latest version you should also update you run.py
Also save the file log with DEBUG level file_level=logging.DEBUG

@SiIason
Copy link
Author

SiIason commented Jan 29, 2021

New pull Initial Observations:
I had 2 unclaimed rewards showing, I started the script and it collected one.
The second reward remained uncollected and no new time was being recorded.
I manually collected the second reward and time started to be recorded.
However its showing as recording two rewards again: https://prnt.sc/xtli8a

@Tkd-Alex
Copy link
Owner

However its showing as recording two rewards again: https://prnt.sc/xtli8a

Yes It's correct. Also, if you use Twitch you will see the same reward multiple times.
+50% Proton Boost x3 , +50% XP Boost x3 , +100% Proton Boost x2
https://www.twitch.tv/drops/campaigns?dropID=c30f52c2-77b8-49eb-9ebb-0479590a4aa8
image

@SiIason
Copy link
Author

SiIason commented Jan 29, 2021

Your right but that's not what I was referring to.
I meant that it was showing two separate rewards going up at the same time.

https://prnt.sc/xtm3b4

@Tkd-Alex
Copy link
Owner

OK, you are right. There is something wrong here 🤣

@SiIason
Copy link
Author

SiIason commented Jan 29, 2021

Stuck again now : https://prnt.sc/xtmwf6
Restarting the script did not collect anything this time.

Will wait for you to update before continuing.

Tkd-Alex added a commit that referenced this issue Jan 29, 2021
…uired check the drop in inventory and get the dropInstanceID for auto-claim #21

Signed-off-by: Alessandro Maggio <alex.tkd.alex@gmail.com>
@Tkd-Alex
Copy link
Owner

@SiIason . New commit ready for test 🔥

@SiIason
Copy link
Author

SiIason commented Jan 29, 2021

Ermmm: https://prnt.sc/xtqci5
I don't know if this is as expected?

Had to manually collect the two outstanding drops before anything happened.
When I collected the first it started the timer again, which I think is right.
When I collected the second, it started two times as you see in the screenshot.

@SiIason
Copy link
Author

SiIason commented Jan 29, 2021

I'm going to watch normally to see if I can get them back to normal.

@Tkd-Alex
Copy link
Owner

I think this time It's correct.
image

@SiIason
Copy link
Author

SiIason commented Jan 29, 2021

Weird they aren't queued up on each other though, its like double/triple rewards for time watched.
I'm currently building up 1x 100% reward and 1x 200% reward for the same time watched.

Down to 1 timer on the 200% reward. Maybe it was all as planned? Maybe the initial script set something wonky?
Maybe... your beating the system ;p

@SiIason
Copy link
Author

SiIason commented Jan 29, 2021

Are you able to check if any items are ready to claim before 'watching' for new items. (at script startup)
This appears to bung up the process, until they are manually cleared.

@Tkd-Alex
Copy link
Owner

Down to 1 timer on the 200% reward. Maybe it was all as planned? Maybe the initial script set something wonky?
Maybe... your beating the system ;p

LOL. I think It's only a different claim system for this game. Nothing else.

Are you able to check if any items are ready to claim before 'watching' for new items. (at script startup)
This appears to bung up the process, until they are manually cleared.

Do you mean to claim all pending reward on startup?

@SiIason
Copy link
Author

SiIason commented Jan 29, 2021

Do you mean to claim all pending reward on startup?

Yes

@SiIason
Copy link
Author

SiIason commented Jan 29, 2021

Assuming we're all good here :) a few quality of life ideas. Would you like me to submit these as improvments?

  1. Are you able to check if any items are ready to claim before 'watching' for new items. (at script startup)
    This appears to bung up the process, until they are manually cleared.

  2. Restrict checking for drops to one stream, as additional stream dont count towards time watched for rewards. Also you dont know which stream is gaining rewards and which isn't.

  3. If we watch 1+ streams manually does that override the 1-2 streams the script is monitoring, in which case if would be handy to check for those claims as well as those monitored by the script (periodically)?

@Tkd-Alex
Copy link
Owner

  1. Are you able to check if any items are ready to claim before 'watching' for new items. (at script startup) This appears to bung up the process, until they are manually cleared.

This is OK for me. I'll do as usually as argument, so a user can choose to claim all drops on startup or not.

  1. Restrict checking for drops to one stream, as additional stream dont count towards time watched for rewards. Also you dont know which stream is gaining rewards and which isn't.

I know which streamer s gaining rewards because I have the channel_id for each 'drop-progress' type. So I don't understand why and what I should restrict. I mean, I don't check anything 🤣 I've only the periodic update from WebSocket. Or you would suggest sending game name in request only for one streamer at once? It's possible that I have not understood you. My English It's not the best

  1. If we watch 1+ streams manually does that override the 1-2 streams the script is monitoring, in which case if would be handy to check for those claims as well as those monitored by the script (periodically)?

If I understand correctly this is related to the 2. ? Assume my script stop sending the WatchMinuteEvent but we are listening for drop-progress topics (via WebSocket). In this case If you watch manually a stream we will receive the event and when current time is equal to required time we will go claim the drop.

@SiIason
Copy link
Author

SiIason commented Jan 29, 2021

Or you would suggest sending game name in request only for one streamer at once?

This. Yet to see multiple games be rewarded at the same time.
But, I assume if you have two manual Twitch windows open Twice does the same, so maybe leave as it?

Tkd-Alex added a commit that referenced this issue Jan 29, 2021
Signed-off-by: Alessandro Maggio <alex.tkd.alex@gmail.com>
@Tkd-Alex
Copy link
Owner

yes, leave as it for the moment

Seillama pushed a commit to Seillama/Twitch-Channel-Points-Miner-v2 that referenced this issue Oct 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants