MRepo repository bugs

[jason-eu]

MRepo version; 1.5.0-alpha02.3aff200

On this version of the app, it is easy to copy/paste/share the repository links because the feature exists.

On this version of the app the following repo works just fine;

https://apt.izzysoft.de/magisk/

The following repo throws a bug;

https://ya0211.github.io/magisk-modules-alt-repo/

MRepo Version 2.0.0-beta01.06e13e8

On this version of the app, it is IMPOSSIBLE to share the repository links because the feature has been removed, effectively rendering the app inferior to its previous version as it lacks this feature.

On this version of the app the following repo throws a bug;

https://apt.izzysoft.de/magisk/

The following repo works just fine;

https://ya0211.github.io/magisk-modules-alt-repo/

[SanmerDev]

This is expected behavior, all repos need to be migrated to new data structure.

As for the sharing function, it should be added.

[Kevinco1]

How is it impossible to copy and paste the address in order to add repos within MRepo? You did that in order to return that error. Also, you were able to add a different repo.....

But we now have an explanation regarding the error thanks to the dev, Izzy has to migrate to new data structure

[mdkcore0]

Got the same issue, is repo-util already stable? if not, MRepo should support both, imho

[upbox-org]

@IzzySoft
Could you please update your magisk repo?

[IzzySoft]

Could you please update your magisk repo?

Currently not (busy on other tasks). And I still wait for the signal by @ya0211 for a stable version being available I should update to (last I remember was a signal to better wait for that). Did I miss that signal? Thanks to the ping here, I NOW see v2.0.0 has been tagged 5 days ago, so I assume that? Are there any "update instructions", as the entire structure was changed – things that one should take care of?

MRepo should support both, imho

It's a good idea to keep the client "backward compatible", as @mdkcore0 already suggested (F-Droid e.g. still supports its old XML index, which was superseded by JSON index-v1 years ago, though it already has even index-v2 for quite a while now), to not "break things"). I had to work around things not yet available back then, so an update will certainly take some time here as I'll need to adjust everything else (e.g. for the website part). It took you several months to establish the new structure (which certainly was some work, I'm definitely not complaining about that part but definitely appreciate it) – not nice to expect repo maintainers to have their part updated on such short notice (or actually having them find out it no longer works), no offense meant.

So sorry folks, guess I'll have to ask you to use an older version of MRepo meanwhile if you want to use my repo 😢 I certainly won't find time for the update within the next week or two. Especially not without knowing in advance the "crucial parts" I have to be aware of when upgrading.

This is expected behavior, all repos need to be migrated to new data structure.

And how would their maintainers know they had to do that? By getting reports from users that it suddenly seems to be broken? Not the best approach I guess 😉

So what are the steps required?

• move config.json from config/ to json/
• run util/cli.py check for the migration
• run util/cli.py index so it takes care for the rest (update modules locally and create the json/modules.json I guess? Or is the new track command now taking care to update the modules, and index only creates/updates the json/modules.json?)
• no errors, so run util/cli.py sync to publish
• done?

There's now index, sync and track, while before sync did the full job. It's great this is now split up (IIRC it was even I who suggested that back then). But which of the 3 updates the local modules? Which syncs to the server? What does the new util/cli.py track do? The Readme doesn't tell. Which means the repo maintainers need to try-and-err. Which I lack the time for currently, sorry 😢

[SanmerDev]

@IzzySoft

So what are the steps required?

• Just add new fields to track.json and run cli.py sync.
• Other migrations will be done by cli.py

I didn't explain how to migrate since util already does most of the things. You only need to use the new cli.py sync or cli.py index (even if you don't add new fields), you can still generate index v1

Backward compatible

It may be implemented, soon No idea

[IzzySoft]

Just add new fields to track.json

OK, that will take a little for the currently 81 modules but shouldn't be too hard.

and run cli.py sync

Which will do what with the new version? If it immediately syncs to the server, that might break things before I've tested and fixed things (see below, my website presentation of the repo might need adjustments, too; I haven't checked yet if at least the new index format is backwards compatible so other tools working with the index could just ignore new fields). Could you also shortly outline what track and index do now, if they are to be run separately or what of their parts are also run by sync (as you mention only to run the latter)?

Other migrations will be done by cli.py

Oh? Going by the Readme, I had thought that would be the task of check:

positional arguments:
  command
    config            Modify config of repository.
    track             Module tracks utility.
    github            Generate tracks from GitHub.
    sync              Sync modules in repository.
    index             Generate modules.json from local.
    check             Content check and migrate.

Maybe you could include some short hints there what the commands are intended for (and what they do)? For example, Module tracks utility. does not really state what it does.

I'll see to it as soon as I can, but I won't be able to even reply here for the next few days. Would be good to have the necessary details available then.

You only need to use the new cli.py sync or cli.py index (even if you don't add new fields), you can still generate index v1

That might work to get MRepo accepting my repo again – but would possibly break my website presence which is not (yet) adapted to the new index format. Same for my framework which is used for the maintenance. So this will need some testing, too – for which I'd need a way to do everything but not have the results sync'd to the server yet (so I can fix things before some "broken stuff" goes live).

Thanks in advance!

[IzzySoft]

PS: I've just checked, and the sync step (to upload stuff to the server) is a separate one handled by my framework, so I can make sure it's disabled for the testing period. Still would need to know the other details, though.

[SanmerDev]

This is the new modules.json(v1): https://github.com/ya0211/magisk-modules-repo-util#v1

cli.py track: As its description, this is a utility tool that helps with adding/deleting/modifying track.json.

cli.py index: It has only one function, which is to generate modules.json. This can be useful when you have manually modified track.json and do not want to execute cli.py sync, only want to update modules.json.

Using cli.py index --version 0 can still generate modules.json for v0.

cli.py check: 1. Check if the module's id matches (zip file, directory name, track.json, update.json). 2. Check if the URL in update.json matches the BASE_URL in config.json. 3. Remove empty values from track.json.

The migration of config.json will be automatically done by the Config class. This means that any subcommand that uses config.json will have the functionality to migrate config.json (github, sync, index, check).

[IzzySoft]

Thanks for the details, Sanmer – much appreciated! Maybe one day we'll have a full documentation so nobody has to dig for those details 😉 Thanks to your input, I today started to see for the upgrade – but immediately ran into some issues. For one, it looks like the current version only works if running from inside a Git tree. If (as in the past) one only copies the util/ directory and the cli.py script, each call just yields

Traceback (most recent call last):
  File "./cli.py", line 26, in <module>
    sys.exit(Main.exec())
  File "/mnt/data/web/ftp/repo/magisk/util/sync/cli/Main.py", line 50, in exec
    parser = Parameters.generate_parser()
  File "/mnt/data/web/ftp/repo/magisk/util/sync/cli/Parameters.py", line 76, in generate_parser
    version=str(get_version_code()),
  File "/mnt/data/web/ftp/repo/magisk/util/sync/__version__.py", line 36, in get_version_code
    count = int(GitUtils.commit_count())
  File "/mnt/data/web/ftp/repo/magisk/util/sync/utils/GitUtils.py", line 58, in commit_count
    return int(cls.exec("git rev-list --count HEAD"))
ValueError: invalid literal for int() with base 10: ''

So 1.5 MB "bare repo" (the .git directory with its contents of all the code history) just to get the number of commits, which are not really needed to use the util? Is that really needed? For a tool that by itself is less than 500 kB? What if someone downloads the ZIP or TAR from releases/, which do not include the .git/ directory? All they will see is above "stack trace".

I've opened MRepoApp/magisk-modules-repo-util#27 for this – and luckily can work around it easily by using a git clone.

That figured, I am now looking what has changed in the syntax:

• cli.py config has completely changed – e.g. --add-module and --remove-module seem to be completely gone. No hint to what replaced them. I guess it wil be the new track. It seems to be there with -a resp. -r, but the parameters must now be named (instead of being simply positional). So I assume instead of cli.py --add-module $module_id $update_to $license $changelog one now has to use cli.py -a id=$module_id update_to=$update_to license=$license [changelog=$changelog], using KEY as it is in the resulting track.json? The -r part is of course rather clear.
• cli.py sync --remove-unused seems to be gone entirely. Luckily I never used that (my framework just had that command already integrated). Is that what --force does now? And cli.py sync without any options does the same job it did before?
• the new cli.py check has interesting options – not crucial for the update now, but still unclear: --check-id: Check id of the module in all json – what for? I guess that they are all "in sync" and maybe also match their directory name? --check-url: Check urls of files in update.json – for reachability I guess? (was answered above, apologies)

Structural changes to modules.json will require some rewrites on my end. As v0 and v1 cannot co-exist, it doesn't help much I could still create a v0 index: the app still would need v1, and both files have the same name. Which means that while I now have a broad overview what needs to be adjusted on my end, it will still need some time for me to implement and test those changes (@Retardium @mdkcore0 @Kevinco1 and @jason-stream – I'm afraid that means it will take a little longer, do not expect it within the next few days).

Further it's a bit disappointing that translation, category and antifeatures did not make it to the index (nor to track.json), so I still have to add these by other means (see MRepoApp/magisk-modules-repo-util#7). Those are fields people know from F-Droid (and my corresponding repo) and expect to be present (with translation admittedly being rarely used), so I will need them.

[IzzySoft]

Oof, that was an extra night-shift now: updated my core library so it can deal with the v1 index now (a quick check with the WebUI showed no remaining issues). The remaining framework still needs to be adjusted (no updates until then), but at least MRepo should no longer complain.

@Retardium @mdkcore0 @Kevinco1 @jason-stream can one of you confirm the repo is loaded correctly again in MRepo?

[mdkcore0]

I confirm it is working now, many thanks @IzzySoft and you all!

[IzzySoft]

Thanks @mdkcore0 – that's good news! Working on the update part then (seems to be mostly running already), so hopefully remaining gaps are closed until the weekend.

[IzzySoft]

For those still looking here: Everything seems to be in working order again now (thanks @ya0211 for your help and your fast solving of open points!). For the "final confirmation" I still need to wait for the next module update coming up and a new module to add, but I'm confident everything has been addressed and, where needed, properly adjusted.

[IzzySoft]

OK, looks like updates are now working fine, too, so I'll close this browser tab now. Thanks for all who helped – reporting, fixing and testing alike!

[BurhanBudak]

Whatever is going on, the app made the camera stop working.
ROM: LOS with MicroG, Magisk

[IzzySoft]

@BurhanBudak I suggest to open a separate issue for that. Please also include additional details such as the device you're using and what Android version it's on (these might help to narrow down what could possibly cause it).