-
Notifications
You must be signed in to change notification settings - Fork 643
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Help wanted - training sets / images and text description #7
Comments
https://www.w3resource.com/python-exercises/web-scraping/web-scraping-exercise-8.php - check this out. This only extracts
Please check if this works. Similarly, I guess you would be able to extract the image description by taking the next |
i wonder if just using the alt might work
there are these data dumps here |
This seems a better 'alt'ernative :P. Have u tried using it? |
no - wont get to it. need help. |
Here is 78k image-caption pairs (3.8 GB zipped) I scraped recently as part of a project. A smaller, ~1k pairs sample of the data (only 58MB, useful for testing) can be found here. I'll take a look at extracting something out of the wikipedia data (I also found this wikimedia archive), but if someone gets to it before me then please do share 😉 there's also probably some stuff that could be extracted from sites like the-eye.eu and archive.org. There's data everywhere, it usually only needs a bit of cleaning. |
I found it helps if you remove all txt files with size zero. vqvae will then point out which images are bad (jpgs that werent jpgs) and you can delete them. also run another filter that will find "odd" images is |
It’s possible we could use Wikipedia extract ~ 40gb offline file and scrape the images / text. Unless ms coco is easier or ???
https://levelup.gitconnected.com/two-simple-ways-to-scrape-text-from-wikipedia-in-python-9ce07426579b
The text was updated successfully, but these errors were encountered: