Use Steam Profile All Games to Skip Already Owned

[cwilliams5]

Is your feature request related to a problem? Please describe.
It's imperfect that the script has to !addlicense for items I already own, is there a better way?

Describe the solution you'd like
I can see the the script builds out a activated_packages.txt for packages it knows are activated. Can't this list be prefilled by scrapping http://www.steamcommunity.com/id/XXX/games/?tab=all and taking any appIDs from there as already owned?

Describe alternatives you've considered
N/A

Additional context
For someone who has already used other methods to add packages and has a lot, the first run of this program is a super long time !addlicense for 10k+ items I already have. Could be mostly skipped by prefiltering owned items via steam profile.

[cwilliams5]

If it helps this is how I pull appIDs off my profile in a PHP script I have for a different purpose:

    $html = file_get_contents('http://www.steamcommunity.com/id/XXX/games/?tab=all');

    // function to extract string data between 2 landmark points
    // http://www.plus2net.com/php_tutorial/php_string_extract.php
    function my_strip($start,$end,$total){
        $total = stristr($total,$start);
        $f2 = stristr($total,$end);
        return substr($total,strlen($start),-strlen($f2));
    }

    // extract the javascript containing all games and game data
    $contents = my_strip("var rgGames = [","];",$html);

    // explode this string out to an array based on javascript seperator
    $games = explode(",{", $contents);

[Luois45]

The idea itself is good and an important feature to get implemented.
I've found a way to get all activated steam licenses pretty fast.

1. Download steamcmd (https://developer.valvesoftware.com/wiki/SteamCMD#Downloading_SteamCMD)
2. Paste this script with the USER values replaced into cmd: steamcmd +login YOUR_USERNAME YOUR_PASSWORD +licenses_print +quit > steam-license-export.txt
3. Enter your steam authenticator key into the cmd. It can take up to a few minutes after this.
4. You will have a steam-license-export.txt file.

The packageIDs could be read out of this file easily with a short script.

[Luois45]

I've tested also a simper solution with javascript:

1. Open the steam license website in your browser
2. Paste the following script into the console of your browser

var elements = document.querySelectorAll(
  "div[class='free_license_remove_link'] > a"
);
var games = [];
for (let i = 0; i < elements.length; i++) {
  var game = elements[i].href;
  game = game.match(/([1-9])\w+/g)[0];
  games.push(game);
}
console.log(games);

3. The script prints all currently activated licenses to the console

[Luois45]

One important aspect is that no very complex solution is needed because the script does create its own list with the already activated licenses.

[Luois45]

If it helps this is how I pull appIDs off my profile in a PHP script I have for a different purpose:

    $html = file_get_contents('http://www.steamcommunity.com/id/XXX/games/?tab=all');

    // function to extract string data between 2 landmark points
    // http://www.plus2net.com/php_tutorial/php_string_extract.php
    function my_strip($start,$end,$total){
        $total = stristr($total,$start);
        $f2 = stristr($total,$end);
        return substr($total,strlen($start),-strlen($f2));
    }

    // extract the javascript containing all games and game data
    $contents = my_strip("var rgGames = [","];",$html);

    // explode this string out to an array based on javascript seperator
    $games = explode(",{", $contents);

@cwilliams5
The http://www.steamcommunity.com/id/XXX/games/?tab=all site isn't loading when I replace XXX with my account name.
I can see the first few lines of the code after a few minutes.

[cwilliams5]

One important aspect is that no very complex solution is needed because the script does create its own list with the already activated licenses.

Yes but the script is currently attempting 16,512 licenses, sleeping 90sec each. I would wager I have probably 16,400 already on my account. So the amount of time this script has to run doing !addlicense wait 90s for stuff I already have is a big delay. I know the next time I run the script it wont matter because of the activated_packages.txt cache, but I'm talking about prefilling activated_packages.txt ahead of time.

Download steamcmd

Yes but if you incorporate this, then people need to supply your script/program with their steam credentials which a subset of people won't want to. My creds are already in ASF which you are using for the activations. If you use games page, then I would only need to provide you my profile URL

Open the steam license website in your browser

Same issue, have to be logged in. With the games page you don't. That said, I think the games page might be less complete in what it lists (like it might miss DLC or something), I don't know I havent compared it.

The http://www.steamcommunity.com/id/XXX/games/?tab=all site isn't loading when I replace XXX with my account name.
I can see the first few lines of the code after a few minutes.

I think it could be two things.

1. If you try and open in a browser it will hang, at least it does for me. If you have as many licenses as you and I do, I think the HTML here is too heavy for the browser. Try to curl or otherwise download the page without rendering it and you should get it.
2. Are you sure thats your profie? Apologies if you know but your steam account name and profile url are different. Not everyone has a short profile name, or one that matches their account. You can go to your profile in steam and right click copy URL, or go to steamcommunity.com login and see what the URL to your profile is.

EDIT* My statement that games page doesn't require logging in is only true for people who have it set to public. There is a way to make your profile private and obviously this would fail for them.

[Luois45]

Download steamcmd

Yes but if you incorporate this, then people need to supply your script/program with their steam credentials which a subset of people won't want to. My creds are already in ASF which you are using for the activations. If you use games page, then I would only need to provide you my profile URL

I could just adapt the script which is working on the games page, but the reason I created it like it is, is because I do want the script to be added to autostart and run in the background so that it isn't any effort after installation.

Open the steam license website in your browser

Same issue, have to be logged in. With the games page you don't. That said, I think the games page might be less complete in what it lists (like it might miss DLC or something), I don't know I havent compared it.

The http://www.steamcommunity.com/id/XXX/games/?tab=all site isn't loading when I replace XXX with my account name.
I can see the first few lines of the code after a few minutes.

I think it could be two things.

1. If you try and open in a browser it will hang, at least it does for me. If you have as many licenses as you and I do, I think the HTML here is too heavy for the browser. Try to curl or otherwise download the page without rendering it and you should get it.
2. Are you sure thats your profie? Apologies if you know but your steam account name and profile url are different. Not everyone has a short profile name, or one that matches their account. You can go to your profile in steam and right click copy URL, or go to steamcommunity.com login and see what the URL to your profile is.

Yes, it was just my browser trying to load the site.

I've had a look into it and the output shows 4128 games out of the 8375 packages I do have currently activated.
I do think that it would be nice to implement that into the activate_packages.py script directly because it does help a bit in the performance and should be easy to implement.
For people with many packages, I do think that a simple JavaScript script which outputs a string which can simply be copied into the cache file should be enough.

[cwilliams5]

I've had a look into it and the output shows 4128 games out of the 8375 packages I do have currently activated. I do think that it would be nice to implement that into the activate_packages.py script directly because it does help a bit in the performance and should be easy to implement. For people with many packages, I do think that a simple JavaScript script which outputs a string which can simply be copied into the cache file should be enough.

Saving up to 4k out of the 16k is a good speed up, with extra steps for those willing. So I don't know but sounds like the a good path:

1. Add an option in config generation to put a steam profile URL, use it to prefill the activated packages cache.
2. Add a quick section to readme explaining how to use javascript on licenses page to prefill activated packages cache more thoroughly.

[Luois45]

I've had a look into it and the output shows 4128 games out of the 8375 packages I do have currently activated. I do think that it would be nice to implement that into the activate_packages.py script directly because it does help a bit in the performance and should be easy to implement. For people with many packages, I do think that a simple JavaScript script which outputs a string which can simply be copied into the cache file should be enough.

Saving up to 4k out of the 16k is a good speed up, with extra steps for those willing. So I don't know but sounds like the a good path:

1. Add an option in config generation to put a steam profile URL, use it to prefill the activated packages cache.
2. Add a quick section to readme explaining how to use javascript on licenses page to prefill activated packages cache more thoroughly.

I've implemented both.

Edit:
Thanks for the great feature suggestions @cwilliams5

[cwilliams5]

I've implemented both.

I'm testing now. OK both methods are working. Here is something interesting I'll share just because its a curiosity.

I did a set comparison between the IDs from my profile games scrape (A) and those from the javascript on the licenses page (B). I knew B would have more, but to my surprise it looked like this:

Turns out A (the games page scrape) has thousands of items that is NOT in B (the license export). Turns out the best way to prefill your script is to do BOTH methods and take the union. And in doing so, your script has no new licenses for me. But I think thats because your license file is 8 days old. Steam.db shows me a few dozen new ones, but I ran theirs less than 8 days ago. So I'm going to stay away from theirs and I bet I start picking up some in yours. Once you update it. How often do you update it? I look forward to running this with a fresh package update and then seeing if the steam.db tool still sees anything outstanding.

Edit: Thanks for the great feature suggestions @cwilliams5

Thanks for making/distributing/maintaining this and being so responsive. I'm a better idea person than programmer xD

Here's another two ideas to close out this "speed" issue. First, I think there is an improvement to be made with the way you use activated_packages.txt. Here's what you do:

1. Download package_list.txt. Load into array.
2. Shuffle thearray.
3. For each ID
   * Check if its in activated_packages txt
   * If it is, do nothing.
   * If it isn't, activate using ASF and sleep for 90 seconds.
4. Repeat

There are 1 or 2 problems with this. First I don't know Python well enough, but you might be opening a disk file every item. But more to my point here, this does not "show" the user that a certain amount of data is already done.

By that I mean, imagine that out of the 16k items in package_list.txt, I already have 5,000 activated. That means every 3rd or so item will be skipped, but the other 2/3 are still sleeping 90sec. It will not be obvious to me that they program is honoring my activated packages.

I think all of this is addressed and smoother if the logic is changed to this:

1. Download package_list.txt. Load into array.
2. Remove all items in activated_packages from this array.
3. Shuffle thearray.
4. For each ID
   * Activate using ASF and sleep for 90 seconds.
5. Repeat

This would is not only likely be more efficient from a filesystem perspective, but the user would in our example immediately see the progress bar rapidly fill to 33% then start slowly adding items.

As to how to accomplish step 2, i dont know if python has a function or 3rd party library to subtract one array from another. If it does, great, if not, just do another loop before your main activation loop. I don't really know Python but probably something like: Split activated_packages into array activated. For Each activated, packagelist.remove(activated).

OK - Last speed up. You currently sleep for 90 seconds. But 50 activation an hour is one activation every 72 seconds. Call it 74 seconds for safety if you want. Changing the sleep to 74 seconds would cause the program to compete 18% faster, which for a new steam account and your current package list would be 3 days faster.

[cwilliams5]

There are 1 or 2 problems with this. First I don't know Python well enough, but you might be opening a disk file every item.

One pass through the adding process is 33k+ disk accesses, your definitely hitting the disk hard.

[Luois45]

I've implemented both.

I'm testing now. OK both methods are working. Here is something interesting I'll share just because its a curiosity.

I did a set comparison between the IDs from my profile games scrape (A) and those from the javascript on the licenses page (B). I knew B would have more, but to my surprise it looked like this:

Turns out A (the games page scrape) has thousands of items that is NOT in B (the license export). Turns out the best way to prefill your script is to do BOTH methods and take the union. And in doing so, your script has no new licenses for me. But I think thats because your license file is 8 days old. Steam.db shows me a few dozen new ones, but I ran theirs less than 8 days ago. So I'm going to stay away from theirs and I bet I start picking up some in yours. Once you update it. How often do you update it? I look forward to running this with a fresh package update and then seeing if the steam.db tool still sees anything outstanding.

Didn't know that, but will change the instructions for users with many packages accordingly.

I think all of this is addressed and smoother if the logic is changed to this:

1. Download package_list.txt. Load into array.
2. Remove all items in activated_packages from this array.
3. Shuffle thearray.
4. For each ID
   • Activate using ASF and sleep for 90 seconds.
5. Repeat

I will change the logic to this one.

OK - Last speed up. You currently sleep for 90 seconds. But 50 activation an hour is one activation every 72 seconds. Call it 74 seconds for safety if you want. Changing the sleep to 74 seconds would cause the program to compete 18% faster, which for a new steam account and your current package list would be 3 days faster.

I'm not sure if it would be faster because the 50 activations per hour is just a number which is calculated by the community and no official number.

[Luois45]

There are 1 or 2 problems with this. First I don't know Python well enough, but you might be opening a disk file every item.

One pass through the adding process is 33k+ disk accesses, your definitely hitting the disk hard.

I've found that the code of activate_packages.py in line 77 +/- 5 lines triggers pretty heavy disk usage.
I will fix the problem.

[cwilliams5]

OK - Last speed up. You currently sleep for 90 seconds. But 50 activation an hour is one activation every 72 seconds. Call it 74 seconds for safety if you want. Changing the sleep to 74 seconds would cause the program to compete 18% faster, which for a new steam account and your current package list would be 3 days faster.

I'm not sure if it would be faster because the 50 activations per hour is just a number which is calculated by the community and no official number.

While you are right about it not being official, I can tell you from using steamdb tools for YEARS that it is exactly 50 that you can do per hour, and this has been a stable fact for many years.

My PR #14 includes speeding this up with almost 2 minutes of buffer time just in case.

But otherwise this issue here is addressed completely, good work, closing