New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

steam workshop collection downloader #1623

Open
shadow-absorber opened this Issue Sep 20, 2017 · 24 comments

Comments

8 participants
@shadow-absorber
Copy link

shadow-absorber commented Sep 20, 2017

as requested by @dgibbs64 i am posting the script i made to download mod collections from steam workshop.

#!/bin/bash
curl https://steamcommunity.com/sharedfiles/filedetails/?id=1101449362 > source.html
echo "./steamcmd.sh +login "USERNAME" "PASSWORD" +force_install_dir ../serverfiles \\" > moddownload.sh
cat source.html \
| grep -E "<div class=\"workshopItemPreviewHolder " \
| sed 's/"><div class=.*//' \
| sed 's/.*id=//' \
| sed -e 's/^/+workshop_download_item 211820 /' \
| sed -e 's/$/ \\/' >> moddownload.sh
echo "+quit" >> moddownload.sh
chmod +x moddownload.sh
./moddownload.sh
rm source.html
rm moddownload.sh
@shadow-absorber

This comment has been minimized.

Copy link

shadow-absorber commented Sep 20, 2017

this thing should help solve the issue #960

@shadow-absorber

This comment has been minimized.

Copy link

shadow-absorber commented Sep 20, 2017

so what the script does in detail is:
step 1. download the workshop collection's html page
step 2. start creating a bash script to download the mods using steamcmd with the login info
step 3. output the html text in a readable way
step 4. make grep search the html file for the mod ids
step 5. format the ids in a list way for steamcmd in the bash script
step 6. add a line to quit the steamcmd download when its done
step 7. make the scrip executable
step 8. run the steamcmd script to download all the mods and validate their files
step 9. delete the files create to reset the script to before it was run to not cause conflicts when it gets run again

@shadow-absorber

This comment has been minimized.

Copy link

shadow-absorber commented Sep 20, 2017

the thing the script needs is a way to replace the id part in this line:
curl https://steamcommunity.com/sharedfiles/filedetails/?id=1101449362
and also hook into the login for steam thats used when downloading the game
wich is specified currently in this line
echo "./steamcmd.sh +login "USERNAME" "PASSWORD" +force_install_dir ../serverfiles \\" > moddownload.sh

@phit

This comment has been minimized.

Copy link

phit commented Sep 20, 2017

no reason to parse the html, you can get all collection objects with the webapi specifically ISteamRemoteStorage_GetCollectionDetails

example using curl:

curl --data "collectioncount=1&publishedfileids[0]=1101449362" https://api.steampowered.com/ISteamRemoteStorage/GetCollectionDetails/v1/

output example: https://gist.github.com/phit/71cea6c43c7071a40589881966aa9fa4

@shadow-absorber

This comment has been minimized.

Copy link

shadow-absorber commented Sep 20, 2017

thanks that is a huge improvement then probably.
the thing is it still needs to be able to split the mods up into the list for the downloads
and not include the collection number as that could cause heavy issues

so plz if you can improve the script post the improved version and we might be able to get this implemented
@phit

@phit

This comment has been minimized.

Copy link

phit commented Sep 20, 2017

something like this should do, I just cut the first publishedfileid using inverted tail, since the first result will always be the collection, obviously this means you can always only have one collection

you could probably do this way nicer using jq to parse the json, but I don't think adding a dependency for this is really wanted

CMD="./steamcmd.sh +login "USERNAME" "PASSWORD" +force_install_dir ../serverfiles "
DOWNLOADITEMS=`curl -s --data "collectioncount=1&publishedfileids[0]=1101449362" https://api.steampowered.com/ISteamRemoteStorage/GetCollectionDetails/v1/ \
    | sed -n 's/.*"publishedfileid": "\(.*\)",/\1/p' \
    | tail -n +2 \
    | sed 's/^/+workshop_download_item 211820 /' \
    | tr '\n' ' '`
    
echo $CMD $DOWNLOADITEMS
@shadow-absorber

This comment has been minimized.

Copy link

shadow-absorber commented Sep 20, 2017

yeah that could work.
tough i would prefer adding \n at the end of all of the +workshop downloads to make it more readable in output form

@phit

This comment has been minimized.

Copy link

phit commented Sep 20, 2017

the output will never be read by anyone in the end product, so I didn't think it was worthwhile

@shadow-absorber

This comment has been minimized.

Copy link

shadow-absorber commented Sep 20, 2017

for example the output for how i wrote mine by myself before i made the first script.

#!/bin/bash
cd /home/sbserver/steamcmd
./steamcmd.sh +login "USERNAME" "PASSWORD" +force_install_dir ../serverfiles \
+workshop_download_item 211820 850109963 \
+workshop_download_item 211820 821455287 \
+workshop_download_item 211820 731220462 \
+workshop_download_item 211820 882900100 \
+workshop_download_item 211820 945707062 \
+workshop_download_item 211820 958539829 \
+workshop_download_item 211820 729426722 \
+workshop_download_item 211820 729558042 \
+workshop_download_item 211820 729429063 \
+workshop_download_item 211820 729438381 \
+workshop_download_item 211820 764887546 \
+workshop_download_item 211820 764888606 \
+workshop_download_item 211820 796877993 \
+workshop_download_item 211820 730544933 \
+workshop_download_item 211820 734170655 \
+workshop_download_item 211820 730745660 \
+workshop_download_item 211820 729427436 \
+workshop_download_item 211820 947429656 \
+workshop_download_item 211820 947922190 \
+workshop_download_item 211820 774083065 \
+quit
@shadow-absorber

This comment has been minimized.

Copy link

shadow-absorber commented Sep 20, 2017

which is helpful if you have to debug it for some stupid reason

@shadow-absorber

This comment has been minimized.

Copy link

shadow-absorber commented Sep 20, 2017

but then again i guess it would work your way also without problems

@shadow-absorber

This comment has been minimized.

Copy link

shadow-absorber commented Sep 20, 2017

so now with the new api thing the scrip would look like this:

#!/bin/bash
echo "./steamcmd.sh +login "USERNAME" "PASSWORD" +force_install_dir ../serverfiles \\" > moddownload.sh
curl -s --data "collectioncount=1&publishedfileids[0]=1101449362" https://api.steampowered.com/ISteamRemoteStorage/GetCollectionDetails/v1/ \
| sed -n 's/.*"publishedfileid": "\(.*\)",/\1/p' \
| tail -n +2 \
| sed -e 's/^/+workshop_download_item 211820 /' \
| sed -e 's/$/ \\/' >> moddownload.sh
echo "+quit" >> moddownload.sh
chmod +x moddownload.sh
./moddownload.sh
rm moddownload.sh

that would use the api avoiding future problems because of html and should probably work forever
it also makes the resulting temporary file for steam cmd a lot easier to read if something messes up.

@shadow-absorber

This comment has been minimized.

Copy link

shadow-absorber commented Oct 11, 2017

for those wondering this is being worked on to get implemented.....

@Fyb3roptik

This comment has been minimized.

Copy link

Fyb3roptik commented Nov 15, 2017

Any updates? How can I download a map? I am unable to do this :(

@shinji257

This comment has been minimized.

Copy link

shinji257 commented Nov 27, 2017

Does this script work if you set +login to anonymous?

@LoadingSAa

This comment has been minimized.

Copy link

LoadingSAa commented Dec 18, 2017

Has there been anymore work put into this? I wanted to try adapt this for use with an ARMA3 server and was wondering if this has gotten any further?

@shadow-absorber

This comment has been minimized.

Copy link

shadow-absorber commented Dec 19, 2017

@shinji257 the script does not work with anonymous logins
@LoadingSAa there has been some work on it but it seemingly stopped when another person helping me quited the project
and for other games then starbound you would change the game id(in starbounds case 211820) to whatever it is for that game. i also recall that the github wiki has guides for most of the games lgsm supports

@shadow-absorber

This comment has been minimized.

Copy link

shadow-absorber commented Dec 19, 2017

for a link on the project info so far: WIP

@dgibbs64

This comment has been minimized.

Copy link
Member

dgibbs64 commented Dec 19, 2017

Decent effort so far. I have sadly been too busy to take a proper look :(. This is something that we should look at integrating in to LinuxGSM at some point.

1 similar comment
@dgibbs64

This comment has been minimized.

Copy link
Member

dgibbs64 commented Dec 19, 2017

Decent effort so far. I have sadly been too busy to take a proper look :(. This is something that we should look at integrating in to LinuxGSM at some point.

@LoadingSAa

This comment has been minimized.

Copy link

LoadingSAa commented Dec 19, 2017

This is what I have so far for arma3 there is an issue with steam CMD timing out constantly though. I'm trying to find out why.

version1: https://ghostbin.com/paste/uvxf8

this is what I'm trying to make work now:

https://gist.github.com/marceldev89/12da69b95d010c8a810fd384cca8d02a

The above script works for me the only thing I can't get to work is collections.

@phit

This comment has been minimized.

Copy link

phit commented May 13, 2018

steam started stripping whitespace at some point which broke my script, switched to jq as its more reliable

curl -s --data "collectioncount=1&publishedfileids[0]=IDHERE" https://api.steampowered.com/ISteamRemoteStorage/GetCollectionDetails/v1/ \
| jq '.response.collectiondetails[] | .children[] | .publishedfileid' \
| sed 's/^/+workshop_download_item 211820 /' | tr '\n' ' '
@rafaeljardim1660

This comment has been minimized.

Copy link

rafaeljardim1660 commented Dec 9, 2018

this script is for python?

@linonetwo

This comment has been minimized.

Copy link

linonetwo commented Dec 15, 2018

And after that, you should edit the serverfiles/linux/sbinit.config

{
  "assetDirectories": [
    "../assets/",
    "../mods/",
    "../../Steam/steamapps/workshop/content/211820/939132577",
    "../../Steam/steamapps/workshop/content/211820/1507130642"
  ],
  "storageDirectory": "../storage/"
}

or steam/steamapps/common/Starbound/linux/sbinit.config if you are not using LinuxGSM.

{
  "assetDirectories" : [
    "../assets/",
    "../mods/",
    "/path/to/Steam/SteamApps/workshop/content/211820/733063633/",
    "/path/to/Steam/SteamApps/workshop/content/211820/743283539/",
    "/path/to/Steam/SteamApps/workshop/content/211820/729769049/",
    "/path/to/Steam/SteamApps/workshop/content/211820/736472581/",
    "/path/to/Steam/SteamApps/workshop/content/211820/732860513/"
  ],

  "storageDirectory" : "../storage/"
}

Don't forget to check whether this file is a good JSON.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment