Pull from live json feed instead of static github version #12

Open
paulirish opened this Issue Feb 16, 2012 · 25 comments

6 participants

@paulirish
H5BP member

http://caniuse.com/jsonp.php I imagine is updated more frequently than when @Fyrd pushes it to his repo.

That true, alexis?

@Fyrd

Yes, that's right. Updated every time I make a change to the data.

@drublic
H5BP member

Hee,

what do you think about a cron job that curls the file from caniuse.com/jsonp.php, writes the new file, commits and pushes automatically. Or maybe better, on each request we check the age of our data.php and if it's older then a day, we replace it with the version hosted at caniuse.com.

Drawbacks?

@paulirish
H5BP member

the latter sounds great

@drublic
H5BP member

I've created a new branch for this to not pollute my master-branch with more unmerged commits. You can find it over here.

I am requesting the file from http://caniuse.com/jsonp.php, remove the parenthesis as it's no callback-function, write the contents into the new file and that's about it.
I've also added a small logger, which writes into a file, each time the jsonp.php-file is requested.

It looks like we don't need the data.json file anymore, as we can write the requested data from caniuse directly into the cache. This is a bit more performant then opening files twice, I guess.
Maybe I'll look into it later.

cc: @jonathantneal, @nimbupani

@jonathantneal
H5BP member

I have a cron job running on my wikies server that does this, except it might be putting from the original github repo.

So ... uh ... obviously I'm putting my +1 on this! Of course it would make sense to have the cron job run not off my sandbox site :D

@drublic
H5BP member

I think including the cron into the script adds a bit more independence to the repo.
On my local machine I have no cron which requests the latest version of caniuse. We could avoid errors for everyone dealing with the repo by including this fix once.

@skddc

What's the status on this? Unfortunately, the current data is quite outdated at the moment. I wanted to use it for getUserMedia, but all the current browsers are marked as unknown, while updated on caniuse.com.

Should we update manually in the meantime? It's not much work, when you discover something outdated, and I don't think the idea of being so dependent on a single person's data, awesome as he or she might be, is a good one.

@skddc

One more thought: wouldn't it be a preferable situation if @Fyrd would pull data from a GitHub repo for caniuse (where we can all send pull requests) instead of only occasionally pushing it to all of us?

Sorry if I'm missing something or there was already a discussion about this. Didn't really follow these 2 projects before.

@drublic
H5BP member

@skddc I've pushed my changes to a new branch "caniuse-data". Check out this one if you want to use an automatic pull from caniuse.com/jsonp.php.

FWIW the problem with getUserMedia is that caniuse.com/jsonp.php only returns "old" data. I've manually included the latest version of jsonp.php in my local repo. Didn't change anything.
Sorry, but I guess we'll have to wait until @Fyrd updates the file.

@skddc

You see the absurdity of having to wait for someone to update a file, when we're all using GitHub, and it's really just about 3 characters that need to be replaced, don't you?

@drublic
H5BP member

Please feel free to submit a PR which pulls more recent data from the source you can find.
As we build upon caniuse we are depending on the data provided. Sorry for any troubles.

@jonathantneal
H5BP member

The most recent source is always the one on his website? I'll make a cronjob to pull from that right now.

@drublic
H5BP member

@jonathantneal I thought this is the most recent. I've implemented a kinda "pull cron" in the caniuse-data-branch. It's not a cron though :)

@jonathantneal
H5BP member

@drublic , I tried writing a PHP curl script ( source @ https://gist.github.com/2293056 ) that pulled from http://caniuse.com/jsonp.php, but it always timed out. I'm stuck there at the moment.

@drublic
H5BP member

@jonathantneal Did you have a look at what I did? I've implemented this with get_file_contents here. This does not time out for me.

@jonathantneal
H5BP member

@drublic, I've fixed my pull from fyrd's github repo and updated this repo with his changes. That resolves a lot of the problems we were having. However, that still doesn't address the direction we want to take on this ticket.

Who updates the live site?

@nimbupani
H5BP member

The live site is updated from the master every 10 mins.

@drublic
H5BP member

@skddc The data for getUserMedia can now be found here.

@jonathantneal Thanks for fixing this. Which direction do you mean? As I pointed out my fixes pull the jsonp.php-file from caniuse.com and save the new version if it's older than a day. For me this is fixes the issue.
Maybe I just don't get the point…

@skddc

@drublic What does that mean? It's still not live on the main API endpoint, right?

@skddc

The question is still unanswered if it's enough to just complete the ticket by its original name, or if we can/should find a better solution that doesn't rely on a single person to update the data. I'd gladly help keeping it up-to-date, and during our work on @5apps we have to deal with this every day anyway. Would be great if @Fyrd could share his view on the matter.

@drublic
H5BP member

@skddc No, it's not live yet or merged into the master. I want to await the "go" from @jonathantneal, because he has another solution / current state.

It feels like @Fyrd is updating the jsonp-file regularly. So requesting this file seems the best solution to me. That's why I implemented it.

@paulirish
H5BP member

Yes the jsonp file gets updated as the site does. It's the best source for the data.

+1 on site pulling from here every 10minutes.

+1 on waiting for the Go from jon neal.

Hans, let me know if you need access to logs once this is in, or anything.

@Fyrd

Yeah, what Paul said. The jsonp file is updated as part of updating any data-related part of the site.

@jonathantneal
H5BP member

I am in favor of pulling from WHEREVER the jsonp is first updated. :) From the website is fine, and if it changes later, that's fine too. It's plain sense to me.

@drublic
H5BP member

I've changed the time for which the file is cached on our server to 10 minutes as Divya's cron is doing and Paul +1ed.

As I think we are not really finding a solution for this at the moment, I will just keep the stuff I did in the caniuse-data branch. Maybe it comes in handy when discuss this again in the future :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment