List of git repos for font projects #1262

felipesanches · 2017-03-29T14:13:39Z

This pull request addresses the problems described at issue #1241

Extracting the list into a separate file: fontprojects.py (issue fonttools#1241)

…s, though. And some of these are binary only, so they still need to be inspected individually (issue fonttools#1241)

repos are enabled (not commented out) (issue fonttools#1241)

And annotation dozens of other repos with the issues found in each of them. (issue fonttools#1241)

so that we can generate things like this summary: ===================== Number of font repositories in this file: "TTX": 15 "OK": 41 "UFO": 3 "GH-PAGES": 2 "NOTE": 5 "GFONTS-REPO": 1 "OTF": 12 "?": 35 From the total of 114 repos, 46 are enabled. Enabled means: "OK" or "NOTE" status. (issue fonttools#1241)

Number of font repositories in this file: "TTX": 16 "OK": 47 "UFO": 18 "GH-PAGES": 2 "NOTE": 13 "GFONTS-REPO": 1 "OTF": 16 "404-ERROR": 1 "SOURCE-ONLY": 1 From the total of 115 repos, 60 are enabled. Enabled means: "OK" or "NOTE" status. (issue fonttools#1241)

(issue fonttools#1241)

davelab6 · 2017-03-29T20:22:42Z

@m4rc1e what do you think?

m4rc1e · 2017-03-29T20:27:10Z

@davelab6 I'll review this. I'm still using our pipeline doc :-) so I need to crosscheck.

For now, a hardcoded python file is ok. We may need to improve this in the future.

felipesanches · 2017-03-29T20:40:59Z

A git-versioned python list is the most flexible (and comfortable) solution for me. And it can very easily spit out a CSV file, if needed for other purposes.

font-family names, based on the foldernames in the google/fonts repo. We'll still have to fill the gaps in this table... Number of font repositories in this file: "TTX": 18 "OK": 46 "UFO": 17 "GH-PAGES": 2 "NOTE": 13 "GFONTS-REPO": 1 "OTF": 15 "404-ERROR": 1 "?": 802 "SOURCE-ONLY": 1 From the total of 916 repos, 59 are enabled. Enabled means: "OK" or "NOTE" status. (issue fonttools#1241)

Now we have 91 repos enabled. ==== Number of font repositories in this file: "TTX": 15 "OK": 76 "UFO": 18 "GH-PAGES": 2 "NOTE": 15 "GFONTS-REPO": 1 "OTF": 15 "404-ERROR": 1 "?": 777 "SOURCE-ONLY": 2 From the total of 922 repos, 91 are enabled. Enabled means: "OK" or "NOTE" status. (issue fonttools#1241)

something is still wrong about the way we schedule and run worker containers and how we decide to give up or try again upon failure.

m4rc1e · 2017-04-04T11:54:47Z

I've been improving the G Doc but I now agree with @davelab6. This should be a .csv stored in a top level folder such as ./data

davelab6 · 2017-04-04T12:11:40Z

The gdoc is fine for drafting and the CSV URL is therefore good while drafting is ongoing, and can check that file in from time to time as needed

…

On Apr 4, 2017 7:54 AM, "Marc Foley" ***@***.***> wrote: I've been improving the G Doc but I now agree with @davelab6 <https://github.com/davelab6>. This should be a .csv stored in a top level folder such as ./data — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1262 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAP9y7bKuF6xAwqiLP1YZrVxuWyccnJbks5rsi-HgaJpZM4MtCn6> .

feed the container with data from a GoogleDocs spreadsheet. But we keep a cache file locally as well just in case (and for versioning purposes) (issue fonttools#1241)

(issue fonttools#1241)

m4rc1e · 2017-04-04T13:15:48Z

dashboard/containers/dispatcher/fontprojects.py

+#!/usr/bin/env python
+import csv
+import urllib
+CACHE_CSV = "fontprojects.csv"


Why not store this in memory using StringIO?

https://docs.python.org/2/library/stringio.html

I do store the data on disk on purpose. So that I can keep track of history on the git repo. I'm OK with fetching data from a remote spreadsheet, but I also like the guarantee of having access to the data even if the spreadsheet goes offline.

m4rc1e

Overall, a fairly elegant way to get the Google Doc into a .csv, without using gspread. It made me smile.

It would be nice to move this out of the dashboards directory. This functionality is not just related to the dashboard.

m4rc1e · 2017-04-04T13:24:00Z

@davelab6 Felipe's solution uses the Google Doc without the need for fancy dependencies which require authentication. He's cracked it.

m4rc1e · 2017-04-04T13:50:20Z

dashboard/containers/dispatcher/fontprojects.py

+CACHE_CSV = "fontprojects.csv"
+
+def get_repos():
+  handle = open(CACHE_CSV)


You're not closing the file in this function.

I'd opt for this:

def get_repos(): with open(CACHE_CSV) as handle: repos = [] for row in csv.reader(handle): if not row: continue repos.append(row) return repos

If I'm not mistaken, garbage collection does its work of closing the file upon returning the function call, as there's no remaining reference to the file object.

felipesanches · 2017-04-04T14:53:05Z

It would be nice to move this out of the dashboards directory. This functionality is not just related to the dashboard.

There are some limitations imposed by the Dockerfile, which does not allow references to parent folders. So we have to keep the code in the container folder. The only way out of this that I can see would be to split out the dashboard as a separate git repo. There are other considerations to make if we go that way, though.

(issue fonttools#1241)

each font project repo.

(issue fonttools#1241)

otherwise it would re-run forever. (issue fonttools#1241)

(issue fonttools#1241)

Using git commit --depth={MAX_NUM_ITERATIONS} (issue fonttools#1241)

(issue fonttools#1241)

we'll later increase this value again, but now, for testing purposes, let's stick to the master of each font-repo. (issue fonttools#1241)

(issue fonttools#1241)

Cormorant SC, Amatica SC, etc... (issue fonttools#1241)

(issue fonttools#1241)

felipesanches · 2017-04-05T19:02:38Z

I have merged this because it contains fixes to actual fontbakery checks. Off course, expending is an ongoing effort, so we'll probably open more PRs for that later.

m4rc1e · 2017-04-05T21:33:35Z

Thanks

felipesanches added 7 commits March 28, 2017 21:51

working on the full list of font repos.

26a3bd6

Extracting the list into a separate file: fontprojects.py (issue fonttools#1241)

increasing the list to ~100 font repos (still lacks font-file prefixe…

7bc0a96

…s, though. And some of these are binary only, so they still need to be inspected individually (issue fonttools#1241)

make the list of font repos tell us automatically how many

1ae6c2c

repos are enabled (not commented out) (issue fonttools#1241)

Raising active repos to a total of 46.

4ebdba5

And annotation dozens of other repos with the issues found in each of them. (issue fonttools#1241)

comment naming the fields in the git repos list

d0f2423

(issue fonttools#1241)

m4rc1e self-requested a review March 29, 2017 20:25

felipesanches added 6 commits March 29, 2017 20:58

attempting to fix the container orchestration

3f076d9

something is still wrong about the way we schedule and run worker containers and how we decide to give up or try again upon failure.

refactoring the worker container script

1e05bc5

exporting the current list as a CSV file

64a42ca

the CSV file with font project repos

060fcdf

felipesanches added 2 commits April 4, 2017 09:18

removing hardcoded list of repos and adding code to

6b21044

feed the container with data from a GoogleDocs spreadsheet. But we keep a cache file locally as well just in case (and for versioning purposes) (issue fonttools#1241)

updating cached list based on marc's spreadsheet

e6cde91

(issue fonttools#1241)

m4rc1e reviewed Apr 4, 2017

View reviewed changes

felipesanches added 4 commits April 4, 2017 20:43

Update status/font-prefixes

fc4fa77

(issue fonttools#1241)

limit the maximum number of commits that we run fbakery on

8021df0

each font project repo.

add fontprojects.csv to the dispatcher Dockerfile

b587cd5

(issue fonttools#1241)

fix several issues in the worker script

ab0f0bc

felipesanches added 16 commits April 5, 2017 01:07

add pylint cmd on Travis for the worker container script

3c9d5b5

(issue fonttools#1241)

fix font_prefix for Abhaya Libre repo

f8d28bb

(issue fonttools#1241)

fix anaheim & bevan font_prefixes

ab03407

(issue fonttools#1241)

fix cambay font_prefix

d65a2ec

(issue fonttools#1241)

exit value must be always zero in the worker container

41f382a

otherwise it would re-run forever. (issue fonttools#1241)

fix saving stats to database

6978aba

(issue fonttools#1241)

fix crash on the flask app

c8cc74b

(issue fonttools#1241)

A more efficient way of running a limited num of commits!

7cfb506

Using git commit --depth={MAX_NUM_ITERATIONS} (issue fonttools#1241)

fixing Inconsolata font_prefix

d8968da

(issue fonttools#1241)

reducing to a single commit per family for now

638258d

we'll later increase this value again, but now, for testing purposes, let's stick to the master of each font-repo. (issue fonttools#1241)

fixing Overpass font_prefix

3312d49

(issue fonttools#1241)

indexing by familyname instead of giturl

21045fb

add a few more exceptions to name table rules, such as:

0e52a63

Cormorant SC, Amatica SC, etc... (issue fonttools#1241)

Added server logs page

1a6cd61

(issue fonttools#1241)

fix travis pylint rule

b92ec81

add a couple missing dependencies to travis setup

da6ad4b

felipesanches merged commit 94cde5c into fonttools:master Apr 5, 2017

felipesanches mentioned this pull request Apr 26, 2017

Track project metadata: upstream repo, downstream package targets/requests, etc #291

Closed

davelab6 mentioned this pull request Jul 25, 2017

Develop a list of upstream URLs for every family in google/fonts googlefonts/fontbakery-dashboard#33

Closed

davelab6 mentioned this pull request Aug 28, 2017

Document the list of all font project git repos that need to be automatically checked. googlefonts/fontbakery-dashboard#15

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

List of git repos for font projects #1262

List of git repos for font projects #1262

felipesanches commented Mar 29, 2017

davelab6 commented Mar 29, 2017 •

edited

Loading

m4rc1e commented Mar 29, 2017

felipesanches commented Mar 29, 2017

m4rc1e commented Apr 4, 2017

davelab6 commented Apr 4, 2017 via email

m4rc1e Apr 4, 2017

felipesanches Apr 4, 2017

m4rc1e left a comment •

edited

Loading

m4rc1e commented Apr 4, 2017

m4rc1e Apr 4, 2017 •

edited

Loading

felipesanches Apr 4, 2017

felipesanches commented Apr 4, 2017

felipesanches commented Apr 5, 2017

m4rc1e commented Apr 5, 2017

List of git repos for font projects #1262

List of git repos for font projects #1262

Conversation

felipesanches commented Mar 29, 2017

davelab6 commented Mar 29, 2017 • edited Loading

m4rc1e commented Mar 29, 2017

felipesanches commented Mar 29, 2017

m4rc1e commented Apr 4, 2017

davelab6 commented Apr 4, 2017 via email

m4rc1e Apr 4, 2017

Choose a reason for hiding this comment

felipesanches Apr 4, 2017

Choose a reason for hiding this comment

m4rc1e left a comment • edited Loading

Choose a reason for hiding this comment

m4rc1e commented Apr 4, 2017

m4rc1e Apr 4, 2017 • edited Loading

Choose a reason for hiding this comment

felipesanches Apr 4, 2017

Choose a reason for hiding this comment

felipesanches commented Apr 4, 2017

felipesanches commented Apr 5, 2017

m4rc1e commented Apr 5, 2017

davelab6 commented Mar 29, 2017 •

edited

Loading

m4rc1e left a comment •

edited

Loading

m4rc1e Apr 4, 2017 •

edited

Loading