LIBRARIES page

[kattni]

Here is what I would like to see for the LIBRARIES page.

Main page would be three sections:

• CircuitPython Libraries
  • (quick intro to libraries)
• Download Instructions (heading):
  • Embedded links to the latest bundles.
  • Instructions on copying individual libraries to CIRCUITPY drive.
• Contributing (heading):
  • Quick bit about contributing to the libraries
  • New Libraries list (updated by script)
  • Updated libraries list (updated by script)
  • Link to Contributing page.

Contributing page:
Contributing page linked from the LIBRARIES main page will be information aggregated from all the libraries, updated by a script. It will be text blocks interspersed with lists of links to PRs/Issues/Repos (supplied by the script). The formatting will involve section headings, text and links. I would provide text content.

The script:
The script would supply the assets for the lists on the main page and for building the Contributing page. The script will be written by @sommersoft. It will be designed to update the New/Updated libraries lists, and the Contributing page, on a to-be-determined time interval, likely starting with weekly. We would add a cron job to Travis for circuitpython-org to run the script.

For both the lists on the main page, and the content on the Contributing page, the script can run and update a json file which can be used to populate the pages.

Terrible mock-up included.

[kattni]

@jwcooper @tylerdcooper I'd like work together with you to move forward with this as soon as is reasonable. I understand there may be other priorities.

[jwcooper]

@kattni I can start on it this week, I hope.

For the script component (if possible), I'd like to follow the same process we use for the files.json (https://github.com/adafruit/circuitpython-org/blob/master/_data/files.json) circuitpython release process. For libraries, we could simply add a _data/libraries.json file that is pushed to this repository on each release. Then, we can just pull in that data in jekyll, etc.

[sommersoft]

@jwcooper,

Using the same process as files.json shouldn't be an issue. The idea was to have the script in adabot anyway. It will most likely run in a Travis cron job, vs the release builds on circuitpython. Submitting PRs can be the same...

I'll try and get a schema together soon, though it honestly may just come straight out of GitHub's API replies.

[sommersoft]

@jwcooper,

How does the below schema work for you all?

Truncated Raw Example:

[
  {
    "library_updates": {
      "new": {
        "Adafruit_CircuitPython_LIFX": "https://api.github.com/repos/adafruit/Adafruit_CircuitPython_LIFX"
      },
      "updated": {
        "Adafruit_CircuitPython_INA260": "https://api.github.com/repos/adafruit/Adafruit_CircuitPython_INA260",
        "Adafruit_CircuitPython_RA8875": "https://api.github.com/repos/adafruit/Adafruit_CircuitPython_RA8875"
      }
    },
    "open_issues": {
      "Adafruit_CircuitPython_SK9822": [
        "https://api.github.com/repos/adafruit/Adafruit_CircuitPython_SK9822/issues/1"
      ],
      "Adafruit_CircuitPython_SI4713": [
        "https://api.github.com/repos/adafruit/Adafruit_CircuitPython_SI4713/issues/2"
      ],
      "Adafruit_CircuitPython_VEML6070": [
        "https://api.github.com/repos/adafruit/Adafruit_CircuitPython_VEML6070/issues/7"
      ],
      "Adafruit_CircuitPython_BluefruitSPI": [
        "https://api.github.com/repos/adafruit/Adafruit_CircuitPython_BluefruitSPI/issues/9",
        "https://api.github.com/repos/adafruit/Adafruit_CircuitPython_BluefruitSPI/issues/8",
        "https://api.github.com/repos/adafruit/Adafruit_CircuitPython_BluefruitSPI/issues/4"
      ]
    }
  }
]

I can change it up if need be. I have more experience reading JSON than encoding...

[sommersoft]

Ha! And in true fashion, now I notice that its using the API URL for library updates. Easy fix.

[jwcooper]

The formatting looks great other than maybe dropping the outer array (unless there will be multiple library_updates and open_issues sections)?

Should be quite easy to get that displayed with liquid. I'll use that as the test schema, and let you know if I need any other changes.

Thanks!

[sommersoft]

I don't imagine there will be multiple update/issues section. Will drop the outer array.

[sommersoft]

I'm going to get an initial PR for /_data/libraries.json in tonight, hopefully. That will at least start things.

But, as I'm constructing the adabot auto-update-majik part, I have a question. This kind of ties into my lack of knowledge with the plan for automating files.json updates.

@kattni, @tannewt:
Should I plan to utilize adabot's fork for cloning, updating, and then PRing? Or would direct commits be preferred?

The main difference I see between the two JSON updates is:

• files.json is based on core releases, and will presumably be run by the core's Travis on those triggers. Or, will it be more schedule based to update the download counts?
• libraries.json will be a cron based update. The cron can be set up on any of the repos. The "easiest" would be to simply add it to the current adabot daily cron (can still be limited to weekly inside the python script).

[jwcooper]

@kattni, @sommersoft
Should we display the open issues in the contributing section since they are included in the libraries.json? If so, it might be useful if we can include the title in the libraries.json with the url (I'm assuming this will be in the API already for github, if not, no big deal).

Also, can we add the release bundle download links in the libraries.json? I can pull them from the API, if we need to, but we're already building this json file, so it may make sense to include it (faster page loads as well, if we don't' have to wait for an API request/response in js).

Possible updated example file:

{
  "releases": [
    {
      "2.x": "https://github.com/adafruit/Adafruit_CircuitPython_Bundle/releases/download/20190414/adafruit-circuitpython-bundle-2.x-mpy-20190414.zip"
    },
    {
      "3.x": "https://github.com/adafruit/Adafruit_CircuitPython_Bundle/releases/download/20190414/adafruit-circuitpython-bundle-3.x-mpy-20190414.zip"
    },
    {
      "4.x": "https://github.com/adafruit/Adafruit_CircuitPython_Bundle/releases/download/20190414/adafruit-circuitpython-bundle-4.x-mpy-20190414.zip"
    }
  ],
  "examples": [
    "https://github.com/adafruit/Adafruit_CircuitPython_Bundle/releases/download/20190414/adafruit-circuitpython-bundle-examples-20190414.zip"
  ],
  "library_updates": {
    "new": {
      "Adafruit_CircuitPython_LIFX": "https://github.com/adafruit/Adafruit_CircuitPython_LIFX"
    },
    "updated": {
      "Adafruit_CircuitPython_INA260": "https://github.com/adafruit/Adafruit_CircuitPython_INA260",
      "Adafruit_CircuitPython_RA8875": "https://github.com/adafruit/Adafruit_CircuitPython_RA8875",
      "Adafruit_CircuitPython_CharLCD": "https://github.com/adafruit/Adafruit_CircuitPython_CharLCD",
      "Adafruit_CircuitPython_FeatherWing": "https://github.com/adafruit/Adafruit_CircuitPython_FeatherWing"
    }
  },
  "open_issues": {
    "Adafruit_CircuitPython_BluefruitSPI": [
      {
        "https://github.com/adafruit/Adafruit_CircuitPython_BluefruitSPI/issues/9": "read_packet needs to check for leftover data in self._buffer",
        "https://github.com/adafruit/Adafruit_CircuitPython_BluefruitSPI/issues/8": "ATI command works ok but errors when debug is used - gives Error (id:0xa180)",
        "https://github.com/adafruit/Adafruit_CircuitPython_BluefruitSPI/issues/4": "HID example code"
      }
    ],
    "Adafruit_CircuitPython_Hue": [
      {
        "https://github.com/adafruit/Adafruit_CircuitPython_Hue/issues/2": "PyPi Setup"
      }
    ]
  }
}

[kattni]

@jwcooper The contributing section will only contain a small blurb about contributing and a link to a second page. This is intentional - the library report page will be a lot of information and I want it separate from the main page which is meant to be a basic introduction to/download page for CircuitPython libraries. cpy.org/libraries will have a link to cpy.org/contributing or cpy.org/libraries/contributing which will contain the issues/PRs/repo issues.

[jwcooper]

@kattni That sounds good. I can move that functionality to /libraries/contributing. We'll probably want the issue titles for that page in the json, since we could re-use that functionality on the contributing, if possible.

[sommersoft]

can include the [issue] title in the libraries.json with the url

Yeah, that's easy enough. I can add that tonight.

Also, can we add the release bundle download links in the libraries.json?

The planned schedule to run the script for libraries.json is once a week. The Bundle releases are done daily (if there is an update). It may be better to JS with the API, or simply point to the release page https://github.com/adafruit/Adafruit_CircuitPython_Bundle/releases/latest.

[jwcooper]

@sommersoft Sounds good. I'll likely hard link the /latest page as well as build the release bundle zip links from the api with js.

[jwcooper]

@sommersoft Here is the latest version of the libraries.json file. The PR is nearly ready to go, just have a couple more things to add in.

https://github.com/adafruit/circuitpython-org/blob/79c6a4fdc566c659dbcfd2db030d3ce20028ba15/_data/libraries.json

[jwcooper]

@kattni, @sommersoft Is the repo level issues section automated? If so, I believe we'd need that in the libraries.json file as well.

adafruit/circuitpython#1246

[kattni]

@jwcooper Yes, it will need to be there. I was unclear with my expectation so @sommersoft didn't know the entirety of what I wanted for the contributions page. We discussed it last night and he's sorting how he wants to do it since it now resembles the current adabot script more closely. Sommersoft can explain better what his plans are now.

[jwcooper]

Sounds good, thank you @kattni!

[sommersoft]

@jwcooper,

Here's a current iteration (i reordered it a little): https://github.com/sommersoft/circuitpython-org/blob/new_lib_json/_data/libraries.json

I can push it to your PR if you'd like. Well, once I remember how to do it. 😄

[jwcooper]

@sommersoft At first glance, this looks great. I can copy it over to my fork and integrate the new components that were added.

[jwcooper]

@kattni, @sommersoft I think we just need the count for the pull requests merged at this point (based on latest PR review).

[kattni]

@jwcooper I believe you are correct! There is one more change that needs to be made to the Contributing page, but that's on our end, not yours (change needs to be made to the script).

[sommersoft]

Yeah, it'll just be another key. How does {"merged_pr_count": "n"} sound, @jwcooper?

[jwcooper]

@sommersoft That works for me.

[kattni]

@jwcooper We've updated the script with the rest of the necessary information to complete the libraries page.

I want to find out, how does the page get updated? Is it triggered by the script updating? Or is there something else to it?

I ask because I am considering having the authors/reviewers/newlibs/updatedlibs lists be updated daily as a rolling weekly list. I want to make sure that we're handling that appropriately if we decide to do it - I think it's simply a change on our end for when the script updates, but I don't know that for certain.

[jwcooper]

The site rebuilds on any commit to the master branch of this repository. Whenever the script is updated and the PR is merged, it should trigger a re-build.

I'll merge in the latest changes to the libraries json.

[kattni]

@jwcooper Are you okay with the Contributors and New/Updated Libraries lists being updated daily? The Contributing page would be updated weekly. I ask because updating it daily would increase the amount of activity on the repo, and you seem most likely to notice. It won't require any changes on your end, it's something we'd do, but it would mean daily builds.

[jwcooper]

I think that's fine as long as it's automated.

[jwcooper]

This was completed and merged.