"No such file or directory: 'generator/gpt2/models/model_v5/vocab.bpe'" #39
Comments
|
Yes, I also am getting that error. |
|
I was getting the same error and I used a different sign in now gives me this error: |
|
I am getting that error as well. That file definitely doesn't exist in the directory location that is specified. |
|
I am also getting the error. |
There is 403 error while downloading and this causes errors while initializing the AI Dungeon. |
|
you beat me to it - I think they probably turned off downloading the necessary models. How big are they? |
|
Same here |
|
Looks like Nick updated the download URL - getting "Network is unreachable" now Anyone able to set up a mirror? |
|
@nickwalton how big are the files that need to be hosted? |
|
A recent commit changed the download url in the install.sh file to an IP that returns 403 errors. A commit that was just pushed changed the URL again to https://students.cs.byu.edu/~nickwalt but this site doesn't work either. You can temporarily hotfix it on your end by changing the download_url back to the original, https://aidungeonmodel.s3-us-west-1.amazonaws.com until Nick can sort it out |
|
Files don't seem that big, why not just host them on the git repo? |
|
I think I just fixed this issue. Sorry for the craziness guys. It was just costing INSANE amounts. But should be reasonable now. Let me know if it still doesn't work. |
|
ahh, the apt-get install of aria2 failed because running locally it needed sudo. the errors weren't visible when running the install script. I see now that its 5.8 GiB This is downloaded everytime someone runs this in that notebook, or is there caching? Sorry still downloading right now locally to my machine, haven't retried the notebook. |
|
@nickwalton -- Details, please! What was the cost per game before the fix? The cost now? How many people are playing it? |
|
And is it your personal money, or the university's? |
|
I'm going to make an issue for this problem |
Additionally, the workaround now fails with: |
|
Running into the same issue, resulting in the following message: `AI Dungeon 2 will save and use your actions and game to continually improve AI Initializing AI Dungeon! (This might take a few minutes) Traceback (most recent call last): |
13.34¢ USD per run with current S3 prices, which doesn't sound like a lot, but according to Google, searches for "AI Dungeon 2" are quickly coming close to eclipsing searches for "World of Warcraft" to give you an idea of how much traffic this is getting. It's been republished by a ton of large tech and gaming sites/blogs/news aggregators.
|
The futurism.com article about this project appeared on my google chrome home screen a few hours ago, so that may lead to a larger influx soon. |
|
Turns out fix wasn't good enough. To answer your questions cost was something like 20-30 cents per download. Got 60,000 unique users to the aidungeon.io site and the charges ended up at 15k for just today. Dr. Wingate, the professor of my lab was sponsoring it, but it's gotten past what he can afford from the labs budget so I had to shut off public bucket access till there is a solution. If anyone wants to set up a torrent system or something then I'm happy to support |
|
IPFS may be a way to handle the model distribution |
|
Omg. You guys are amazing. What's a run? If someone selects "restart and run all", is that a new run? I may have done that 40 times today. |
|
i'd gladly help seed any torrents! |
|
Torrents should be fine too. You can feed a magnet link to aria2c to download files in colab. Someone who already has model files just need to register torrent file on some tracker. |
Same, though I suspect Nick probably doesn't want to spend the time re-implementing this download over a bittorrent client inside of a shell script. If it's any consideration, I know OVH does storage at 1.1¢ per GB, which works out to only about 6.38¢/run, which would decrease your costs significantly, but at the levels of bandwidth we're talking about, I'm thinking the most cost effective option would be to host the game as it's own service on a dedicated server with plenty of processing power. |
|
@InfosecRD see above, aria2c can do magnet links |
I'm no expert on the colab platform, but if this is implemented, I'll help seed from my home connection and on my VPS :) |
|
Awesome guys! And I have one week till I graduate so might as well just
finish haha. Though my GPA this semester is a dumpster fire from working on
this so much.
…On Sat, Dec 7, 2019, 11:05 PM p-groarke ***@***.***> wrote:
You could host the files on a free google drive account (15GB), and use a
link generator to get a direct link to it. Will seed torrent for now. Good
luck!
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<#39>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFJNOQDPCBHMTSGHEMML6GLQXSFEBANCNFSM4JXU5OCQ>
.
|
|
I've got the folder shared on my google drive too, this way you can mount the file directly to avoid copying: https://drive.google.com/drive/folders/1XiDD2BD8vLZaJxZpCrNYjscpvnD3EYrP |
Brilliant idea, Nick might in fact use his own GDrive to have more control over his models. |
Out of curiosity, would this make the torrent obsolete as far as the progress we made on it? |
|
Wait how would this work? Can't people only mount their own Google drive?
…On Sat, Dec 7, 2019, 11:34 PM Portaluke ***@***.***> wrote:
mount the file directly to avoid copying
Brilliant idea, Nick might in fact use his own GDrive to have more control
over his models.
Out of curiosity, would this make the torrent obsolete as far as the
progress we made on it?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<#39>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFJNOQD73FF4P6YIIAPAZBLQXSINXANCNFSM4JXU5OCQ>
.
|
|
They'd have to click the link manually first to get it as a shared folder under their drive, and then they mount their own drive. |
I did this in the colab website. |
|
This error is misleading, because it simply fails to bind to IPv6 port, but it was downloading fine over IPv4. |
|
I think the Google drive idea might be better in practice |
|
I'm working on the Google drive symlink workaround, should be ready soon. |
|
Google Drive workaround: https://colab.research.google.com/drive/1OjBQe4H4C2s-p4-OeJoXw5DStIjPy2VS |
Why does it copy the files? Doesn't the shared folder work fine? |
No, because this method requires mounting the user's Google Drive. |
|
@WAUthethird So I don't quite understand. Why would you download from your google drive instead of mounting your google drive? |
I'm not quite sure what you mean. It copies the files after the drive has been mounted. |
|
@WAUthethird Shared folders don't take up space, and the colab instance uses a symbolic link not a copy. Just checked my google drive usage on my test account and it hasn't changed. |
Ah, I see. I think I was recalling the instances where I got "Quota Exceeded" on a few files and had to make a copy, which did take up space. I've updated the instructions. |
|
I'm seeing an input/output error attempting to run with the workaround link that has the uncensored version. I think I added it to my drive correctly? |
|
I'll continue to work on getting shards on up github. Probably take a few hours to upload. I think the script that does it is just about ready |
|
anyone able to get the google drive shared folder method working? Also it looks like the notebook reset itself. Things are only cached per session. That's a lot of re-downloading for anyone who wants to use this more than once. Also it looks like all of my github hosted shards are up. I'll write a script to pull them down and piece them together. |
|
In the end Google colab just isn't a great place to host this I'm working
on an app version to fix this but might take a couple days. Once that's
available I'll make a way for people who want to tinker or run locally to
play it.
…On Sun, Dec 8, 2019, 10:28 AM JamesHutchison ***@***.***> wrote:
anyone able to get the google drive shared folder method working?
Also it looks like the notebook reset itself. Things are only cached per
session. That's a lot of re-downloading for anyone who wants to use this
more than once.
Also it looks like all of my github hosted shards are up. I'll write a
script to pull them down and piece them together.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<#39>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFJNOQDRWV5QVCM27TKHBDTQXUVCTANCNFSM4JXU5OCQ>
.
|
|
😬 given the demand I can't imagine an app version would be cheap (assuming you mean web-app), especially since this seems to require a high end GPU. With colab google is basically giving away their product at a loss to get people to use it. Plus also isn't there a state you'd need to keep track of for every session? There's a lot of problems to solve. |
|
Torrents work amazingly well now, with downloads averaging to 2MiB/s per client. @nickwalton perhaps you should update the Colab page with information about the torrent, encouraging users to get it and seed it, and bring back the setup instructions. Unless of course, you are already fully immersed into making an official web app, in which case, i wish the best of luck 👍 Web development is a time-consuming job. |
|
If its a matter of having a server to store and serve files on, may I suggest using Hetzner? they offer fully self managed root access servers. I have had no issue with them on maintenance, and the one time that the server was unstable due to unknown hardware issue, they just swapped out the hard drives to new hardware. You can bid on one of their used systems https://www.hetzner.com/sb At the moment they got a intel i7-2600 server with 2x3 TB Raid 1 capacity, and unlimited 1GB/s traffic for 32.27 Euro a month. They have different distros of linux to choose from, or windows if you want to pay a bit more for the license. This way you got full control over the files themselves. |
I run a crypto currency wallet and pool off of mine. If it can handle that, it can handle your needs :) |
Trust me, no standalone server would be capable of dealing with this demand. We currently have about 1000 torrent leechers, and just calculate how much of that bandwidth would be left to each client. 1000 Mbit/s -> 125 MB/s (best case) -> 125kB/s/client. |
|
Google Cloud Platform has a generous $300 free credit to spend over the course of one year. You could rent a VPS temporarily to use as a CDN. This has the benefit of potentially being hosted in the same datacenter as Google Colaboratory, which could enable very fast local transfers over HTTP. |
They offer cloud service too. Their highest standard one is 35.58 euro a month that has 8 vcpu, 32 GB ram, 240 GB storage, and 20 tb/month. Would be cheaper to scale that up as needed than pay a per instance cost. 15k in one day would have paid for over 450 servers serving a total of 9 PB of data for a month. I don't think you need that much. Cheapest option is 2.99 euro a month, 1 vcpu, 2gb ram 20 gb storage and 20TB/month, so you can pick in that range for what your file hosting section needs. |
Update help text to match actual 'reset', 'restart' behavior
When I go to this page and select "Run all", this is what I get:
The text was updated successfully, but these errors were encountered: