iptv-org
Replies: 43 comments 13 replies
-
|
@BellezaEmporium is setting up a Jenkins server for this. |
Beta Was this translation helpful? Give feedback.
-
I have a Jenkins CI running, created the IPTV index files successfully (i haven't recreated all of the actions we have), and the EPG is currently running but I expect it finishing in a whopping 10+ hours for 2-day-EPG for all of the data we have. However it might not stay forever as i'm currently running on some DO credits, and it may be completely drained in the next months. |
Beta Was this translation helpful? Give feedback.
-
This is probably why Github disabled the repo lol We can probably use the Open Collective funds to pay for a server. There's a bunch of providers <$10/month that don't care too much about CPU usage. |
Beta Was this translation helpful? Give feedback.
-
I had built a tvtv.us scraper in Python that took a schedule in less than 2 seconds, so I believe we can optimize @freearhey 's epg-grabber. I need to look deeply through the code. |
Beta Was this translation helpful? Give feedback.
-
|
We can also split it into several Jenkins jobs and run them across multiple servers. You can have multiple workers that connect to one Jenkins server. I may have some spare capacity I can use on my existing servers (I'd have to check). |
Beta Was this translation helpful? Give feedback.
-
|
@Daniel15 To clarify, at the moment we only collect $6/month through Open Collective:
https://opencollective.com/iptv-org#category-BUDGET So even if all @iptv-org/core members decide to donate their portion of the fund to pay for servers, in a year we will run out of money anyway. |
Beta Was this translation helpful? Give feedback.
-
|
what is the situation please i'm still waiting admin and big masters :( ( Thanks ) |
Beta Was this translation helpful? Give feedback.
-
Everyone is waiting. There is no ETA, the new CI system will be done when it's done. For now, you need to be patient. |
Beta Was this translation helpful? Give feedback.
-
If it's $6/month recurring, that should be enough for a server that can run a CI system (if the rest of the team are OK with that). We can also ask members if they'd be kind enough to do recurring monthly donations specifically to cover server expenses. |
Beta Was this translation helpful? Give feedback.
-
Well, if it so. We only need to decide who will administer the server (will it be you or @BellezaEmporium). Then those of @iptv-org/core who want to support this initiative will just have to leave a comment here and I will update iptv-org/ledger to make it clear exactly how much money is available. |
Beta Was this translation helpful? Give feedback.
-
I have multiple servers and am willing to host what we may need. Please let me know if that will be ok with all. If not then you can use my earnings as payment for one. |
Beta Was this translation helpful? Give feedback.
-
|
I have tried to optimize the work time by modifying a few things in the grabber and made available 2 different versions : the "fast" version uses parallel threading linked to Promises, so everything launches in the same time. However, certain links may not be scraped as they're either not fullfilled or answering at time. Another one, the "master / classic" version, is the same implementation with a little optimization test. Both of them have updated dependencies. In the latest testings (through CI) :
I still wonder how the EPGs inside GH Actions used to be to perform that well without having any EPG issues. |
Beta Was this translation helpful? Give feedback.
-
Are these all in the same process? Something else you could try is splitting the work across multiple processes. Have a coordinator process that spins up several worker processes and use some form of IPC to communicate between them. The coordinator could split up the work and pass it to the workers. Doing that, plus using parallel requests, should speed it up a lot. |
Beta Was this translation helpful? Give feedback.
-
Separate CI jobs |
Beta Was this translation helpful? Give feedback.
-
|
I'm not that familiar with the build process so apologies if I'm talking rubbish here. From what I recall we had a number of processes to grab the and update a single database of programme details then once those had finished, a separate process to select the db and build the xml output. My experience of that sort of flat-file databases is that they are ok for smaller project but become a major resource hog when they get beyond a certain size. We are no longer grouping outputs by country, every xml output file only contains programmes from a single source. So do we still that single huge programme db? Would the build be more efficient if each grabber process used it's own (smaller) database. Each xml output could then be created as part of that grabber process rather than building all xmls together at the end. You still have the issue of building the guide list but that can be done using a smaller db that is updated by each of the grabber/build processes. That db only needs one record for each xml showing source, language, number of channels and URL - essential the tables we already have on the homepage. I'd probably add last update date/time to each row as I think that would be useful. |
Beta Was this translation helpful? Give feedback.
-
|
Any luck setting up those Jenkins nodes? |
Beta Was this translation helpful? Give feedback.
-
So far I prefer setting my node as a slave one, letting someone who's better in using Jenkins set up a master node for this particular program. |
Beta Was this translation helpful? Give feedback.
-
|
Will be watching this space for any update |
Beta Was this translation helpful? Give feedback.
-
|
May be watching it for a long time then 😂 |
Beta Was this translation helpful? Give feedback.
-
|
Is there a way to download the source code of the EPG grabber and run it locally? I don't need all those channels so I guess I could run a couple of JS parsers for the channels I need, while I wait for the system to come up again |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
Seeing this a bit late, sorry. Not that we need everyone to do so, but we need a few servers that are capable of grabbing the EPGs and giving the end result into the repo. |
Beta Was this translation helpful? Give feedback.
-
|
Goodness me this is taking longer than I hoped |
Beta Was this translation helpful? Give feedback.
-
|
Hey, It's slightly altered because I actually store the EPG in a database for some data processing, but here's how I do it with the good old ✨PHP✨ and cloud-init: Then I just run a cron job every day My code is quite recent, I actually have been trying it since yesterday, so it doesn't circumvent every eventuality, and I have some unknown things I'm still trying to debug (such as when I put multiple clusters, it seems like it's putting the wrong programs associated with the wrong channel and lang, I don't know if it's in my parsing or my storing, but I haven't seen that issue with max_clusters=1) Maybe that'll help! :) EDIT: Edited because it was, in fact, not working due to a wrong parsing of the file name LOL |
Beta Was this translation helpful? Give feedback.
-
|
Sounds quite convincing, i'm seeing a webhook too. I suppose it's to send the guide after it's complete. It needs to stick up to the CI afterwards, sounds like your code would work in GitLab. |
Beta Was this translation helpful? Give feedback.
-
|
So far on my side, it's working as expected. EPG (for a specified number of networks) is running fine, will still need to fix up some EPGs to make them work again. I'll progressively add a few others when i'll have time. |
Beta Was this translation helpful? Give feedback.
-
|
@iptv-org/core What if we reactivate GitHub Actions in the repository but with self-hosted runners? 
If I understand correctly, in this case the load on the GitHub servers will be minimal, as it will only perform the load-balancer function. And if someone would like to support the project he can do it simply by installing this runner on his machine. Did I miss anything? Or is it better to leave everything as it is? |
Beta Was this translation helpful? Give feedback.
-
@freearhey GitHub said to me that we can't do this. The same limitations apply for self-hosted runners, for whatever reason. This is the part of the terms of use that they flagged us for:
|
Beta Was this translation helpful? Give feedback.
-
|
@Daniel15 I see, then of course the question is off the table. |
Beta Was this translation helpful? Give feedback.
-
|
Since for 4 months no replacement for GitHub Actions has been found and continuing to maintain and develop repositories without basic automation becomes too difficult, I decided to re-activate GitHub Actions. For now, it will only be used to check playlists for syntax errors and to update public playlists daily (iptv-org/iptv#13290). In spite of this, both link checking and EPG parsing will continue to be disabled until a new system is found. So if anyone is still planning to address these issues, welcome! |
Beta Was this translation helpful? Give feedback.
-
Yeah... Sites often have a clause in their Terms of Service stating that you can't scrape their data, and they'd own the copyright to the data being scraped. It's definitely a much larger risk compared to simply linking to video streams. |
Beta Was this translation helpful? Give feedback.
-
Travis CI is pretty much straight-forward as far as I know, having a few tests in there. Azure Pipelines is a complete solution, going from simple code hosting to CI/CD functionalities. |
Beta Was this translation helpful? Give feedback.
-
I don't know. I don't have much experience with any of them. |
Beta Was this translation helpful? Give feedback.
-
|
@freearhey I don't know much about GitHub Actions... where is the config stored? I can see if I can move them across to a different service if I get some free time. |
Beta Was this translation helpful? Give feedback.
-
|
@Daniel15 for iptv-org/iptv here: https://github.com/iptv-org/iptv/tree/master/.github/workflows And in iptv-org/epg all Actions are still disabled. |
Beta Was this translation helpful? Give feedback.
-
Description
One of the conditions of GitHub reinstating this repo is that we can't use Actions any more. We need to find a new build/CI system.
We can look at hosted systems like CircleCI and Travis, or host our own Jenkins server.
Beta Was this translation helpful? Give feedback.
All reactions