-
-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove ability for youtube scraper to create multiple ZIM (one per playlist) #878
Comments
What do you want to do with these recipes which have the playlist mode enabled?
|
@benoit74 Let us go over them later today and revert back. |
Add |
How should we move forward on this? Should we wait for you to recreate all recipes, or can we simply delete them and you will recreate them on the fly (deleting the recipe will not remove the ZIMs anyway)? Do you need a configuration export so that it will be easier to recreate? Do you plan to create all recipes manually? What do you mean by "except Khan, zenus and ruangguru" ? Do you plan to simply delete these recipes and not create anymore ZIMs for these ones? |
Ok this I had misunderstood. If there is a way to easily export/duplicate the recipes then by all means let's do it, but otherwise we'll need to recreate them manually (and then only delete the original ones). Khan, zenius and ruangguru will be deleted entirely (recipes AND zim files). |
OK, thank you. Export is easy, it will just be a "raw" copy of the configuration, just so that you have a reference of the old configuration. E.g. for aimhi_playlists (I redacted the secrets), I will export the configuration below and then delete the recipe:
Is this useful? Duplicate is something you already have with the "Clone" button. But you still have to input everything else. From my PoV, this last remark emphasis that:
I don't know if we should live with it, try some quick and dirty wins on some of these topics, or implement a real solution. |
It is probably for @RavanJAltaie to decide how she wants to proceed, but I'm not sure the export is really useful as neither her nor myself have the skills to create the new recipes via script. Our last discussion was to clone existing recipes (in which case (I'd delete them after the deed is done). As for next steps, the quick and dirty tends to be somewhat permanent in this house and not exactly convenient for the non-dev end user either: I suggest we park it until this becomes a real project. |
OK, so next steps before I can start to work on this issue are:
Correct? Note that I'm not speaking about the deletion of unwanted ZIMs, since there is no dependency AFAIK and we can do it at any time, at your own convenience I'm waiting for your GO to perform the last step which consist in removing the ability to use the "playlists mode" in Zimfarm |
I realize that @RavanJAltaie was not on the thread and missed that part. I've assigned her now so she can confirm to you when all new recipes have been created |
Now I'm confused, recipes with multiple playlists are ok? the only problem is deactivating playlist mode? @benoit74 |
@RavanJAltaie Yes, you are right. Recipes with multiple playlists in one ZIM are OK. |
and yes we just want to get rid of the playlist mode |
All fixed successfully! |
Great, thank you! I reopen the issue because I still have my part of the job to do (remove ability to create youtube recipes which will create multiple ZIMs at once) |
@RavanJAltaie I'm sorry but |
It's not clear from this ticket what happened exactly and what will happen:
|
All fixed. |
Again, I still have my part of the job to do |
@RavanJAltaie could you please detail recipe per recipe of #878 (comment) what has been done? I had a quick look and it seems that in many cases, you simply removed the playlist mode and created one big ZIM instead of many small ones, is this correct? The only exception is madrasa? When you used this "create only one ZIM instead of many small ones" approach, it looks like you kept the old small ZIMs in the library, is this intentional? Content is evergreen so we do not mind to keep them in the library and not update them anymore? I'm not convinced by this strategy, usually there was only 5/6 playlists and it did not looked like the number of playlists was frequently updated. Small ZIMs are usually more practical for our users. For https://farm.openzim.org/recipes/voa_learning_en_all for instance, we moved from ZIM ranging from 59.48 MB to 12.72 GB to one enormous (from my perspective at least) 24.93G ZIM. But maybe users are always downloading all ZIMs, so the extra work to create individual ZIMs is not worth it. It is just that this decision is very opaque and has not been explained, so it feels a bit weird. For madrasa I'm not convinced about the ZIM name / filename. For instance you choose And for madrasa is there any reason to keep the two disabled recipes? Especially madrasa_ar_playlists which still uses the playlist mode? |
Yes that's correct, this is the decision made by @Popolechien & me after discussing the #878 issue.
That's the strategy followed in creating madrasa playlists, but for the few corrected playlists, we've decided to keep them in one file, but I can re-discuss this with @Popolechien today and change it if agreed upon. Personally I don't think it worths splitting the playlists.
I agree with you, I'll change the naming for all the files and apply this on new creations as well.
No, no reason, I'll open an issue to delete them. |
Also, as the convention clearly expresses, Project name instead of domain name should be exceptional. I have the feeling this rule frequently abused. @Popolechien @RavanJAltaie please clarify this |
in this case the naming for madrasa should be: Youtube_ar_madrasa_astronomy? |
|
We must reserve
No need to discuss it again if it has been agreed upon, just it would have been better to put these conclusions here before so that everyone involved would be aware of this and we keep a track record, I'm pretty sure we will have a question about it in few months. What about older ZIMs (per playlist), do we keep them in the library? For madrasa, since you are changing the name, you will probably also have to delete older ZIMs. |
Let's Everything else can be discussed separately if needed (and is at least partially already an ongoing effort) |
See openzim/youtube#147.
Recipes using this configuration should be listed and a migration scenario should be decided first.
The text was updated successfully, but these errors were encountered: