Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add importer for Divipola datasets #16

Merged
merged 3 commits into from Sep 7, 2017

Conversation

Projects
None yet
2 participants
@c-w
Copy link
Member

commented Sep 6, 2017

See #14 for background information on this pull request.

Try it live: http://13.72.77.67/features/name/alto%20de%20lisboa

After running the importer, we have just under 6k new features in the database.
Most of these features come from the centro-poplado datasets and therewith
represent small towns or cities.

features=# select count(*) from features;
 count
 -------
   5879

features=# select count(*), layer from features group by layer order by count desc;
 count |  layer
-------+----------
  4754 | locality
  1093 | county
    32 | region

Unfortunately the Divipola data doesn't give us much additional information on
top of just the name, type and geometry of the features, so additional fields
like population will not be populated in the featureService.

Additionally, this PR also contains fixes for two bugs that were exposed by the
Divipola dataset: lowercasing differences between Postgres and NodeJS and
handling of queries for places with names that contain commas.

@c-w c-w requested review from timfpark and erikschlegel Sep 6, 2017

@c-w c-w self-assigned this Sep 6, 2017

@jcjimenez
Copy link
Contributor

left a comment

LGTM

c-w added some commits Sep 6, 2017

Add importer for Divipola data
After running the importer, we have just under 6k new features in the database.
Most of these features come from the centro-poplado datasets and therewith
represent small towns or cities.

```
features=# select count(*) from features;
 count
 -------
   5879

features=# select count(*), layer from features group by layer order by count desc;
 count |  layer
-------+----------
  4754 | locality
  1093 | county
    32 | region
```

Unfortunately the Divipola data doesn't give us much additional information on
top of just the name, type and geometry of the features, so additional fields
like population will not be populated in the featureService.

Resolves #14
Handle case-insensitive comparison of diacritics
NodeJS and Postgres differ in how they treat characters like diacritics
when lowercasing so we should always lowercase both sides of our
equality check in the same language.

An example to illustrate the differences:

```
+-------------+------------------------------+----------------+
| Environment | Operation                    | Result         |
+-------------+------------------------------+----------------+
| Postgres    | lower('BOGOTÁ, D.C.')        | 'bogotÁ, d.c.' |
| NodeJS      | 'BOGOTÁ, D.C.'.toLowerCase() | 'bogotá, d.c.' |
+-------------+------------------------------+----------------+
```

@c-w c-w force-pushed the load-divipola branch from f609ff4 to a47ae44 Sep 7, 2017

@c-w c-w merged commit 503d4b5 into master Sep 7, 2017

@c-w c-w deleted the load-divipola branch Sep 7, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.