Github org/repo abstraction

[uliska]

In and of itself this doesn't do much yet, but it is a step in the direction of #1172, and I would be glad about some early feedback.

It allows to create Repo and Organization objects allowing access to the contents of organizations (so far: list included repositories) and repositories (so far: navigate their tree, get URLs and info about all the objects).

Later there will be facilities to download individual objects or whole downloads.

Communication works through Github API, which will for example also allow the retrieval of download and clone links for managing repositories through Git.

NOTE: I am pretty sure that the networking part of the code has to be made more robust (although the actual exception handling will have to be implemented later in GUI parts using the module).

For testing: If you add the following code at the end of frescobaldi_app/app.py and start Fresocbaldi from the command line you should see the relevant output, you can also play around with the strings to see other organizations or repositories.

# TEMPORARY FOR TESTING!
def test_github():
    import download
    import github
    import github.repo
    import github.org

    print("Organization: frescobaldi")
    try:
        org = github.org.Organization('frescobaldi')
        print()
        print(org.repository_names())
        repos = org.repositories()
        for repo in repos.values():
            print(repo['name'] + ':')
            print(repo['description'])
            print()
    except (
        download.ServerException,
        download.NotFoundException,
        github.JSONException
    ) as e:
        print()
        print("An exception has occured while retrieving info about the organization")
        print(e.__class__)
        print(e)
        sys.exit()


    print()
    print("Deliberatly wrong URL")
    try:
        github.Github.get_json('https://api.github.com/repos/foo/bar')
    except download.NotFoundException as e:
        print(e)

    print()
    print("Deliberately invalid URL")
    try:
        github.Github.get_json('https://api.github.com/repos/openlilylib-resources/lilypond-notation-fonts/git/trees/a8297f8df850b874cb8eaab24a92af397143c35')
    except download.ServerException as e:
        print(e)

    print()
    print("Deliberately fetching invalid JSON")
    try:
        github.Github.get_json('http://lilypond.org')
    except github.JSONException as e:
        print(e)

    print()
    print("Load an actual repository")
    try:
        repo = github.repo.Repo('openlilylib-resources/lilypond-notation-fonts')
    except (
        download.ServerException,
        download.NotFoundException,
        github.JSONException
    ) as e:
        print()
        print("An exception has occured instantiating the repo connection")
        print(e.__class__)
        print(e)
        sys.exit()

    print()
    print("The following notation fonts are available in the repository")
    print(repo.path())
    print()
    fonts = repo.tree().directory('fonts', file=False)
    for f in fonts:
        print('-', f.name())
    print()
    print("These *files* are present in Sebastiano's directory:")
    files = repo.tree().directory('fonts/sebastiano', dir=False)
    for f in files:
        print('-', f.name())
    print()

mainwindowCreated.connect(test_github)

On my PC it resulted in this output:

Organization: frescobaldi

['frescobaldi', 'frescobaldi-themes', 'python-ly', 'python-poppler-qt4', 'python-poppler-qt5']
frescobaldi:
Frescobaldi LilyPond Editor

python-poppler-qt4:
Python binding to poppler-qt4

python-ly:
A Python package and commandline tool to manipulate LilyPond files

python-poppler-qt5:
Python binding to libpoppler-qt5

frescobaldi-themes:
Ready-to-use themes for Frescobaldi

Deliberatly wrong URL
404: URL not found
https://api.github.com/repos/foo/bar

Deliberately invalid URL
HTTP Error 422: Unprocessable Entity
URL: https://api.github.com/repos/openlilylib-resources/lilypond-notation-fonts/git/trees/a8297f8df850b874cb8eaab24a92af397143c35

Deliberately fetching invalid JSON
Invalid JSON retrieved from Github
Error: Expecting value: line 1 column 1 (char 0)
URL: http://lilypond.org

Load an actual repository

The following notation fonts are available in the repository
openlilylib-resources/lilypond-notation-fonts

- gonville
- lilyboulez
- lilyjazz
- lv-goldenage
- paganini
- profondo
- sebastiano

These *files* are present in Sebastiano's directory:
- font-sampler-Sebastiano.pdf
- FONTLOG.txt
- OFL-FAQ.txt
- OFL.txt
- sebastiano.zip

[uliska]

PS: First use cases for this will be listing (and then also downloading/installing) notation fonts and frescobaldi extensions.

[wbsoft]

Nice! Would this pave the way to editing music on github, download openlilylib, have shared snippet repository etc?

Isn't there maybe a ready to use python github api implementation? (Forgive my ignorance🙂)

Just a side note: I often experiment by running python3 in the frescobaldi_app directory and then import app; app.instantiate() and then fiddle with a new module (reloading it using imp.reload(module) each time I change something).

[uliska]

Would this pave the way to editing music on github, download openlilylib, have shared snippet repository etc?

This will initially allow

• downloading notation fonts
• downloading openLilyLib (packages)
• downloading Frescobaldi extensions

(see #1172)

I haven't thought about a shared snippet repository, but that should work too.

Shared music editing will probably not benefit from this because that will be better dealt with in th vcs module. (Somewhere in the back of my mind I still have the vision of something like a Google-Docs like interface with live updates via a remote Git repository ...)

The github module is basically there for getting stuff from Github repositories without Git being installed. The download module is doing the actual downloading stuff and can later also be used for non-github-related downloades, e.g. downloading LilyPond installers or having an extension download sources (e.g. images from a library) etc.

The "client" UIs (e.g. the music fonts dialog, the extensions Preferences page etc.) will have to provide interfaces to select (or detect) whether to interact with the remote resources through Git or the HTTP downloads. For example, it is of course better to clone an extension's repository and have it updated with Git than downloading and unpacking a static snapshot archive file. But we shouldn't require users to work with Git, not even transparently.

One more (minor) thing I thought of is changing the exception handler to not send an email but create an issue on the Github tracker. But that would also involve the authentication issue, and maybe it would be better to simply have the "new issue" page opened for the user and the relevant information automatically copied to the clipboard then ...

---

One thing I found out already is that the Github API is limited to 60 hits per hours for unauthenticated access. Maybe this will be an issue because there are multiple requests necessary for most basic things (e.g. one for the basic info, one more for getting the file tree, one for getting the releases list the one for each release and one for each release's assets list etc.)
I don't know if there's a clean way to change that to authenticated requests without security implications - that would raise the limit to 5000 requests per hour.

Isn't there maybe a ready to use python github api implementation?

There is a package PyGithub, but I'm not sure if it really helps that much, and it would of course add an additional dependency.

OTOH I'm wondering whether it would be wise to make use of Qt for the download module. I haven't looked into it but I suspect that Qt provides useful tools for this kind of tasks.

[wbsoft]

Nice!
Concerning the bug email: If we're able to detect double issue reports then automated posting to the issue tracker is a good idea. I often get the same bugreport many times 😁

(Frescobaldi 1 had a LilyPond downloader and installer. Maybe we can resurrect some of that code.)

[uliska]

Where can I find the code for F1?

Yesterday I did a little research and decided that I'll definitely use the QtNetwork module rather than plain Python. It's a prime case for using a framework.

Regarding bug detection this should be possible - at least Atom can do that, although I don't know how they determine that.

[wbsoft]

Where can I find the code for F1?

F1 is now at wbsoft/lilykde, the old LilyPond download dialog here. It heavily used the KDE libs, such as KIO ;-)

[uliska]

Thanks, I'll see when I can get to that. I must admit that my initial attempt at integrating QtNetwork failed - without errors but also without results. I'll have to check whether that maybe was related to some restrictions in the public WiFi I was sitting back then ...

Just one remark: @wbsoft when you update a comment I won't be notified.