Source updates, usage tagging #110

stepps00 · 2018-05-18T22:13:51Z

Also includes a few minor tweaks to the README, docs.py and source json files. Includes:

Added mz:is_current and edtf:deprecated property descriptions to the README. Similar descriptions to what is found in the properties repo repo.
Updated the docs.py script to include logic to write out "This source has been deprecated." if an edtf:deprecated property has been set.
Added a mz:is_current and edtf:deprecated property to addr, geom, lbl and name json files, as these are not sources (they're catalogued in the properties repo)
Added description to lbl json, for consistency
Crawled the repo and appended values to the usage:* properties for each json file in the repo. Running the makefile added the usage tags to each source in the sources/README file
- There may be additional work to do here. If a usage tag was missed, prefer merging and submitting a follow-up PR.
Adds src:via property to three json sources. These are lists of sources, the key being the source name, value being the source website (not all are included yet).
Updated the docs.py to write out a list of each "via" source in the README (see Quattroshapes).
Updated template file to include new potential properties.

Note that most of the json files have large diffs. This is due to all of the property keys being sorted when writing back to disc. The properties are now alphabetized - I can wire this into the makefile if helpful.

nvkelso

Say when it's been deprecated, please.

nvkelso · 2018-05-18T23:20:15Z

README.md

@@ -40,6 +40,10 @@ While a source .json file in the `whosonfirst-sources` repository does not requi

 * `"description":` A one to two sentence description of the source (_string, optional property_).

+* `"mz:is_current":` Represents whether of not a source is currently in use (_string, optional property_). `0` signifies "not current".
+
+* `"edtf:deprecated":` Indicates the date when a place was determined to be invalid, was never a "going concern" (_string, optional property_). _Format: YYYY-MM-DD (though these dates can be encoded with any valid EDTF syntax)_.


Source, not place

nvkelso · 2018-05-18T23:24:11Z

bin/docs.py

@@ -52,6 +52,10 @@
        if details.get('description'):
            docs.write("_%s_ \n\n" % (details['description']))

+        if details.get('edtf:deprecated'):
+            if not details['edtf:deprecated'] == 'uuuu':
+                docs.write("* %s\n" % ('This source has been deprecated.'))


Also Indicate the date, please.

nvkelso · 2018-05-21T16:52:42Z

Also, for these that were deprecated (addr, geom, lbl and name), indicate in their JSON description that they are properties tracked in the properties repo instead – provide a link to over there.

nvkelso · 2018-05-21T17:04:23Z

I'm not seeing usage for YS, please make sure you're looking for any src:* instead of a list of specific sources like src:geom. This should be a usage for value of a property, in this case label centroids.

nvkelso · 2018-05-21T18:12:08Z

Should we be tracking what placetypes are covered by this source?

Other sources without usage:

acgov
acme
arg-caba
atgov
bebrus
btv - was this Vermont neighbourhood polygons?
can-databc - indicate this is used for clip shorelines for Vancouver neighbourhood geoms?
can-vicodc
chgov
dial (expecting concordance)
dk-geodk
ds (expecting concordance)
ebc (is this British Columbia, Canada constituencies?!)
edtf - we should indicate this is a WOF-ism to indicate "when" certain actions were taken.
gec (fips)
geom – shouldn't this say property?
hsgov
ie-gov
iso – why is this also property instead of just concordance?
itu – why is this also property instead of just concordance?
`lacity_oof
lieu
localwiki
mz - has an extra space in the description causing it to not italicize BEGINNING.
meso
- This source includes CC-BY compatible data from the following organizations:
  Please add the CC-BY compatible text as indicated above.
mudf - has an extra space in the description causing it to not italicize END.
- no usage at all?
no-geonorge
nycgov
nycgov_dca
nycgov_dohmh
nycgov_subway
os
osm – this one is important to say usage concordances!
out – I assume because it's only used in venues repos?
qs - look at all those sources of sources, yeah!
- We monkeyed with this in person to set context, source name, and optional link, desc
se-lant
sfac
sfgov_rbl
smcgov
statcan
svn-sma
thelist - this is venues only?
transitland - this is venues only?
uscensus
uszcta - probably postalcodes?
ys

stepps00 · 2018-05-21T18:46:03Z

#110 (comment) fixed in 4a3c3e3

nvkelso · 2018-05-21T19:05:28Z

src:via has 3 more groups to de-cluster?

Chile: Global Map of Chile: International Steering Committee for Global Mapping / Instituto Geografico Militar de Chile
Palestine: Global Map of Palestine: ISCGM/Palestinian National Authority
South Africa: Global Map of the Republic of South Africa: ISCGM/State Copyright

nvkelso · 2018-05-21T20:25:04Z

source_template.json

@@ -8,8 +8,11 @@
 	"license": "",
 	"license_type": "",
 	"license_text": "",
+    "src:via": {"Whos On First":"https://whosonfirst.org/"},


Please update template for new list of "well known" objects format.

stepps00 · 2018-05-21T21:05:52Z

#110 (comment) taken care of in 9c3ff4f

stepps00 · 2018-05-21T23:14:49Z

9c3ff4f taken care of in #110 (comment)

Notes:

There are several sources not in the whosonfirst-data repo that are catalogued in the whosonfirst-sources repo. This is because the sources are used on postalcodes, venues, or are in a PR/upcoming PR. Notes about the sources you listed above:

Updated usage keys for:

acgov
arg-caba
atgov
can-databc
chgov
ebc
gec
itu
lacity_oof
nycgov_dca
nycgov_dohmh
osm
out
qs
se-lant
sfac
sfgov_rbl
smcgov
statcan
thelist
transitland
uscensus
uszcta
ys

Added usage key, but the source is not yet in use (in PR):

btv
can-vicodc
dk-geodk
ie-gov
svn-sma
no-geonorge

Unable to find concordance, property, or geometry usage for:

acme
can-vicodc
hsgov
lieu
nycgov_subway
os
localwiki
nycgov
bebrus

Deprecated:

dial: should not have been added.. this is the same as itu
ds: in statoids prop, but not in a concordance/prop/geom
edtf: added link to prop repo
geom: added link to prop repo

No change, correct as is:

iso: we store iso:country property and iso concordance.

Updated minor spacing issue:

mz
mudf

Other:

meso: Updated the docs.py to include that text

stepps00 · 2018-05-21T23:26:04Z

#110 (comment) taken care of in 22c6a2b

stepps00 · 2018-05-21T23:48:36Z

@nvkelso - all comments should be addressed in this PR. A few notes:

We have deprecated the name, addr, geom, and lbl property prefixes and pointed to their json files in the properties repo. There are three other properties catalogued, though.. osm. edtf, and lieu. I suggest keeping osm because it is used as a concordance value in some records, but deprecating lieu and edtf and pointing to their json files in the properties repo. Thoughts?
Follow-ups are being tracked in Follow-up standardizing, consistent info with wof-data records #98

nvkelso · 2018-05-21T23:56:10Z

I’m fine with that.

…

On May 21, 2018, at 16:48, Stephen Epps ***@***.***> wrote: @nvkelso - all comments should be addressed in this PR. A few notes: We have deprecated the name, addr, geom, and lbl property prefixes and pointed to their json files in the properties repo. There are three other properties catalogued, though.. osm. edtf, and lieu. I suggest keeping osm because it is used as a concordance value in some records, but deprecating lieu and edtf and pointing to their json files in the properties repo. Thoughts? Follow-ups are being tracked in #98 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

nvkelso · 2018-05-22T23:31:45Z

bin/docs.py

+            docs.write("\n  This source includes `CC-BY compatible` data from the following organizations:\n")
+            for via in details['src:via']:
+                if via['source_note']:
+                    docs.write("  \t* **%s**: [%s](%s) - %s\n" % (via["context"],via["source_name"],via["source_link"],via["source_note"]))


Can you add a test for a link and only add the markdown link decorations if there is a link, please?

Here and one down.

nvkelso · 2018-05-22T23:35:56Z

For Yerbashapes can you add src:via similar to Zetashapes:

Flickr
Mapzen

nvkelso

Great work! 🎉

stepps00 added 3 commits May 18, 2018 14:11

first pass at adding all flags to src jsons

07d7827

makefile run

e16baf6

deprecated sources, add logic to docs.py

4c5f0cb

stepps00 self-assigned this May 18, 2018

stepps00 requested a review from nvkelso May 18, 2018 22:13

stepps00 added 5 commits May 18, 2018 16:59

updated three sources, links, docs.py update

fa5de9e

more docs.py logic, makefile run to update src via links

a82fbcc

formatting to add new line in README

bd28f55

src:via description to readme

b62b225

update template

c3d0476

nvkelso requested changes May 21, 2018

View reviewed changes

stepps00 added 4 commits May 21, 2018 11:22

src:via work, json formatting

293e1cf

docs.py logic for writing src:via and deprecated date

f57f9fd

README updates

087f17b

property prefix links to prop repo

4a3c3e3

stepps00 added 2 commits May 21, 2018 13:13

more usage tags for geometry

9c3ff4f

makefile run

6d623e6

nvkelso reviewed May 21, 2018

View reviewed changes

stepps00 added 3 commits May 21, 2018 15:52

update usage tags, logic in docs py

f2f003f

quotemark

db8e294

makefile run

f65864f

stepps00 added 2 commits May 21, 2018 16:18

update template file

6119686

another makefile run

d58d008

QS updates, make all

22c6a2b

formatting in readme

476f005

stepps00 added 2 commits May 21, 2018 17:31

deprecated two sources

3de1238

make file run

332210a

nvkelso mentioned this pull request May 22, 2018

Create records for all sources during for the mesoshapes import #39

Closed

stepps00 added 8 commits May 22, 2018 10:15

deprecated localwiki source file

22ee8c9

mapzen-derived, docs.py logic

bf53b42

comments in docs.py, swap derived w associated

878ce4e

meso json updates for all src:via:

e411ad8

update mapzen.json

9241624

make all run

a228c1e

wapo link update, make all

2c47feb

update frgov

dca8d82

nvkelso reviewed May 22, 2018

View reviewed changes

stepps00 added 5 commits May 22, 2018 16:39

yerbashapes description update and src:via property

ebda4f2

make all, updated meso and mapzen, template, and readme

12630c5

write links in markdown in docs.py

3439603

statoids words

2193eda

make all run

83045cb

nvkelso approved these changes May 23, 2018

View reviewed changes

nvkelso merged commit d24c4c9 into master May 23, 2018

nvkelso deleted the stepps00/source-updates branch May 23, 2018 00:19

This was referenced May 25, 2018

Add 'source_data' and 'alt' properties #66

Closed

Verify the license terms for the following sources #92

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source updates, usage tagging #110

Source updates, usage tagging #110

stepps00 commented May 18, 2018 •

edited

Loading

nvkelso left a comment

nvkelso May 18, 2018

stepps00 May 21, 2018

nvkelso May 18, 2018

stepps00 May 21, 2018

nvkelso commented May 21, 2018

nvkelso commented May 21, 2018

nvkelso commented May 21, 2018

stepps00 commented May 21, 2018

nvkelso commented May 21, 2018

nvkelso May 21, 2018

stepps00 May 21, 2018

stepps00 commented May 21, 2018

stepps00 commented May 21, 2018

stepps00 commented May 21, 2018 •

edited

Loading

stepps00 commented May 21, 2018

nvkelso commented May 21, 2018 via email

nvkelso May 22, 2018

nvkelso commented May 22, 2018

nvkelso left a comment

Source updates, usage tagging #110

Source updates, usage tagging #110

Conversation

stepps00 commented May 18, 2018 • edited Loading

nvkelso left a comment

Choose a reason for hiding this comment

nvkelso May 18, 2018

Choose a reason for hiding this comment

stepps00 May 21, 2018

Choose a reason for hiding this comment

nvkelso May 18, 2018

Choose a reason for hiding this comment

stepps00 May 21, 2018

Choose a reason for hiding this comment

nvkelso commented May 21, 2018

nvkelso commented May 21, 2018

nvkelso commented May 21, 2018

stepps00 commented May 21, 2018

nvkelso commented May 21, 2018

nvkelso May 21, 2018

Choose a reason for hiding this comment

stepps00 May 21, 2018

Choose a reason for hiding this comment

stepps00 commented May 21, 2018

stepps00 commented May 21, 2018

stepps00 commented May 21, 2018 • edited Loading

stepps00 commented May 21, 2018

nvkelso commented May 21, 2018 via email

nvkelso May 22, 2018

Choose a reason for hiding this comment

nvkelso commented May 22, 2018

nvkelso left a comment

Choose a reason for hiding this comment

stepps00 commented May 18, 2018 •

edited

Loading

stepps00 commented May 21, 2018 •

edited

Loading