Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wikimedia #21

Open
tbo47 opened this issue Aug 9, 2017 · 10 comments
Open

Wikimedia #21

tbo47 opened this issue Aug 9, 2017 · 10 comments

Comments

@tbo47
Copy link

tbo47 commented Aug 9, 2017

Wikimedia has a rss/atom feed of a best picture in there catalog every day. It would be nice to use it as a feed of wallpapers!
https://commons.wikimedia.org/wiki/Commons:Picture_of_the_day

@ifl0w
Copy link
Owner

ifl0w commented Aug 10, 2017

This would be possible quiet easy but I just suggested a Generic JSON API source in #22 where you also could add your opinion if you want to. Since the Wikimedia picture would be an XML response we would have to think of another solution but it would be also possible to add a Generic XML API source.

@lens0021
Copy link

lens0021 commented Jun 16, 2023

Wikimedia Commons also has its own API.

What I've found

I tried the next sources, but failed:

It seems $.parse.images[0] $.parse.images and $.parse are null, while$ is an object. Any ideas?

@Lucki
Copy link
Contributor

Lucki commented Jun 23, 2023

The following works:

  • Request URL:
    https://commons.wikimedia.org/w/api.php?action=parse&format=json&page=MediaWiki%3AFfeed-potd-transcludeme&prop=images&formatversion=2
    
  • URL prefix:
    https://commons.wikimedia.org/wiki/Special:Redirect/file?wptype=file&wpvalue=
    
  • JSON Path:
    $.parse.images[0]
    
References

https://upload.wikimedia.org/wikipedia/commons/a/a0/Geraldine_Ulmar_in_Gilbert_and_Sullivan%27s_The_Mikado.jpg

{
  "parse": {
    "title": "API",
    "pageid": 0,
    "images": [
      "Geraldine_Ulmar_in_Gilbert_and_Sullivan's_The_Mikado.jpg"
    ]
  }
}

Edit:

  • I missed the date part, will update the request URL above later.
  • I missed that 58c3270 wasn't backported to the soup file so in the current version the above solution will fail because of a missing user agent.

@Lucki
Copy link
Contributor

Lucki commented Jun 26, 2023

This is actually a bit more involved.

  • The reason that pasting the normal URL doesn't work is that we're encoding the URI.
    An already encoded URI will be encoded a second time.
    %5B%5BFile%3A%255B%255BFile%253A

  • But if you're trying to decode the URI and use that it won't work either because we're using encodeURI() which will get us a partially encoded URI.

    The characters on the second line are characters that may be part of the URI syntax, and are only escaped by encodeURIComponent().

    [[File:%5B%5BFile:

Possible fixes:

  • Use encodeURIComponent() instead. This may cause problems for URLs that don't want these encoded.
  • Ditch the encoding in the JSON adapter altogether and let the user put in the correct URL.

@ifl0w Any opinions?

@ifl0w
Copy link
Owner

ifl0w commented Aug 15, 2023

@Lucki I'd not remove the encoding entirely since I'd assume that this could break existing setups. If the encodeURIComponent() solution fundamentally works, we could try that.

Another somewhat hacky approach could also be to simply call decodeURI before calling endodeURI. This should not have an effect on decoded URIs and I guess would work around this issue here?

@Lucki
Copy link
Contributor

Lucki commented Aug 15, 2023

I'm sorry, we can't use encodeURIComponent() because that'll also encode https:// and path separators.

URL c&p from Firefox:                   https://commons.wikimedia.org/w/api.php?action=parse&format=json&text=%5B%5BFile%3A%7B%7BPotd%2F%7B%7B%23time%3AY-m-d%7C%7Cen%7D%7D%7D%7D%5D%5D&prop=images&contentmodel=wikitext&formatversion=2
decodeURI:                              https://commons.wikimedia.org/w/api.php?action=parse&format=json&text=[[File%3A{{Potd%2F{{%23time%3AY-m-d||en}}}}]]&prop=images&contentmodel=wikitext&formatversion=2
decodeURIComponent:                     https://commons.wikimedia.org/w/api.php?action=parse&format=json&text=[[File:{{Potd/{{#time:Y-m-d||en}}}}]]&prop=images&contentmodel=wikitext&formatversion=2
encodeURI(decodeURI):                   https://commons.wikimedia.org/w/api.php?action=parse&format=json&text=%5B%5BFile%253A%7B%7BPotd%252F%7B%7B%2523time%253AY-m-d%7C%7Cen%7D%7D%7D%7D%5D%5D&prop=images&contentmodel=wikitext&formatversion=2
encodeURI(decodeURIComponent):          https://commons.wikimedia.org/w/api.php?action=parse&format=json&text=%5B%5BFile:%7B%7BPotd/%7B%7B#time:Y-m-d%7C%7Cen%7D%7D%7D%7D%5D%5D&prop=images&contentmodel=wikitext&formatversion=2
encodeURIComponent(decodeURI):          https%3A%2F%2Fcommons.wikimedia.org%2Fw%2Fapi.php%3Faction%3Dparse%26format%3Djson%26text%3D%5B%5BFile%253A%7B%7BPotd%252F%7B%7B%2523time%253AY-m-d%7C%7Cen%7D%7D%7D%7D%5D%5D%26prop%3Dimages%26contentmodel%3Dwikitext%26formatversion%3D2
encodeURIComponent(decodeURIComponent): https%3A%2F%2Fcommons.wikimedia.org%2Fw%2Fapi.php%3Faction%3Dparse%26format%3Djson%26text%3D%5B%5BFile%3A%7B%7BPotd%2F%7B%7B%23time%3AY-m-d%7C%7Cen%7D%7D%7D%7D%5D%5D%26prop%3Dimages%26contentmodel%3Dwikitext%26formatversion%3D2

No matter how we encode and decode - it's never matching the correct URL copied from Firefox.

@lens0021
Copy link

I do not understand the problem exactly, but an api call https://commons.wikimedia.org/w/api.php?action=parse&format=json&pageid=18176574&prop=images&formatversion=2 is also available (18176574 is the page id of MediaWiki:feed-potd-transcludeme) . Does this solve something?

@Lucki
Copy link
Contributor

Lucki commented Aug 16, 2023

For this particular case that's a workaround, yes.

The problem is for other sites where it's not this easily possible. A general solution would be better.
Also, this prevents pasting a URL from your browser address bar which is quite an inconvenience.

@tbo47
Copy link
Author

tbo47 commented Aug 16, 2023

Here is an example on how to find the latest picture in plain js:

const https = require('https');
const fs = require('fs');

const resolution = process.argv.at(2) || '1280px' // 1024px 2560px

const url = 'https://commons.wikimedia.org/w/api.php?action=featuredfeed&feed=potd&feedformat=atom';

const match1 = 'href="https://commons.wikimedia.org/wiki/Special'
const match2 = 'typeof="mw:File"'

const getUrl = (url) => {
    return new Promise((resolve) => {
        https.get(url, resp => {
            let data = ''
            resp.on('data', c => data += c)
            resp.on('end', () => {
                resolve(data)
            })
        })
    })
}

const download = (url, dest) => {
    return new Promise((resolve) => {
        const file = fs.createWriteStream(dest);
        https.get(url, resp => {
            resp.pipe(file);
            file.on('finish', () => file.close(resolve))
        })
    })
}

(async () => {
    const data = await getUrl(url)
    const urls = data.split(/\n/).filter(l => l.includes(match1)).map(l => l.split(/"/)[5])
    const lastUrl = urls.slice(-1)[0]
    const data2 = await getUrl(lastUrl)
    const picUrl = data2.split(/\n/).filter(l => l.includes(match2)).map(l => l.split(/"/)[7])
    const picName = picUrl[0].slice(11)
    const picFullUrl = `https://upload.wikimedia.org/wikipedia/commons/thumb/5/50/${picName}/${resolution}-${picName}`
    download(picFullUrl, '/tmp/wikimedia.jpg')
})()

@HarrySidhuz0008
Copy link

<script src="https://code.jquery.com/jquery-3.6.0.min.js"></script>
<script>
    // Function to fetch the RSS/Atom feed
    function fetchPictureOfTheDay() {
        // URL to Wikimedia Commons' Picture of the Day feed
        const feedUrl = "https://commons.wikimedia.org/w/api.php?action=featuredfeed&feed=potd&feedformat=atom";

        // Make an AJAX request to fetch the feed
        $.ajax({
            url: feedUrl,
            type: "GET",
            dataType: "xml",
            success: function(data) {
                // Parse the XML feed
                const imageLink = $(data).find("enclosure").attr("url");
                
                // Set the fetched image as the wallpaper
                $("#wallpaper").css("background-image", `url(${imageLink})`);
            }
        });
    }

    // Call the function to fetch the Picture of the Day
    fetchPictureOfTheDay();
</script>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants