⚡ 🚸 Re-enable populating addresses in trips and make sure that they are used on the phone #937

shankari · 2023-07-14T17:58:27Z

Right now, we don't fill in address information in the trips when we process them. Instead, the phone "fills in" the addresses when it receives the trips by making calls to OSM nominatim. This is annoying because we make a lot of potentially redundant calls to nominatim (although @JGreenlee) is caching some information on the phone, and it is also annoying for users because the addresses fill in one by one by one very slowly and it uses up people's data plans.

Instead, while we are filling in place information, we should fill in the addresses so that they will be part of the trip and used in multiple places without having to keep re-looking up the information. It will also have the benefit that we can know the address at the time the trip was taken, rather than the time the trip was viewed, in case there are changes to the map in the middle.

There is existing code to fill this in using nominatim on the server, but it runs into nominatim usage limits https://github.com/e-mission/e-mission-server/blob/8761f4afce635bc4cc7ff1732d9f00956cb5c4ad/emission/analysis/intake/cleaning/clean_and_resample.py#L250

However, we are in the process of signing up for a paid nominatim service. To prepare for that, we should:

make sure that the nominatim code still works (has not bitrotted). We can use the free nominatim service to check this, as long as we only make a few calls an hour (https://operations.osmfoundation.org/policies/nominatim/)
then, work with @JGreenlee to ensure that the stored values are used when possible, and we only make calls from the phone when the server does not send data, which should only happen for draft trips.

shankari · 2023-07-14T18:01:34Z

So concretely:

read and understand the emission.net.ext_service.geocoder.nominatim package
experiment with it directly (using notebook/python REPL) and some arbitrary lat/lon locations
add an integration test for it in emission/tests so we can quickly check if the format changes in some way
load raw data from emission/tests and run the pipeline with nomimatim configured and make sure that the addresses are filled in

After those are done, we can move to the phone changes.

shankari · 2023-07-15T05:18:50Z

For the record, emission.net.ext_service.geocoder.nominatim currently also uses google and falls back to nominatim if Google is not configured. We should remove the Google API implementation - we will only use nominatim going foward.

https://github.com/e-mission/e-mission-server/blob/8761f4afce635bc4cc7ff1732d9f00956cb5c4ad/emission/net/ext_service/geocoder/nominatim.py#L17

nataliejschultz · 2023-07-18T05:07:33Z

I have spent most of my time on this project so far reading and testing the functionality of nominatim.py, reading up on the nominatim api documentation, and preparing to make API calls.

I was initially unsure how to run the nominatim calls locally, since their documentation is not great. I was able to download a docker container from nominatim-docker that contains everything needed to run a localhost and make nominatim calls:

docker run -it \
  -e PBF_URL=https://download.geofabrik.de/europe/monaco-latest.osm.pbf \
  -e REPLICATION_URL=https://download.geofabrik.de/europe/monaco-updates/ \
  -p 8080:8080 \
  --name nominatim \
  mediagis/nominatim:4.2

I then modified nominatim.json to say:

"query_url" : "http://localhost:8080/nominatim/"

per this exchange.

When trying to run, I kept getting the following error:

google maps key not configured, falling back to nominatim
http://localhost:8080/nominatim
>>> reverse_geocoded_json = eco.Geocoder.get_json_reverse("39.4818456", "-106.0445699")
<urllib.request.Request object at 0x100bb5940>
>>> print(reverse_geocoded_json)
{'error': 'Unable to geocode'}

I decided to look through old files since @JGreenlee showed me how (thank you!!!) and found out that I could just use
"query_url" : "http://nominatim.openstreetmap.org" to run my small query. My call worked after that.

shankari · 2023-07-18T06:14:06Z

I was initially unsure how to run the nominatim calls locally, since their documentation is not great.

I'm glad you figured this out - just to clarify:

We can use the free nominatim service to check this, as long as we only make a few calls an hour (https://operations.osmfoundation.org/policies/nominatim/)

so you didn't have to run the nominatim service locally - just make calls from your laptop to the free nominatim service (nominatim.openstreetmap.org)

wrt

>>> reverse_geocoded_json = eco.Geocoder.get_json_reverse("39.4818456", "-106.0445699")

Do you know why it didn't work? Hint: Look at the data that you downloaded into your container.

shankari · 2023-07-18T18:10:55Z

Answer: because we loaded data from Monaco but were trying to access a location in Colorado.
To verify, you can:

use a lat/lon location in Monaco, and/or
load data for Colorado and retry (https://download.geofabrik.de/north-america/us/colorado.html, e.g. https://download.geofabrik.de/north-america/us/colorado-latest.osm.pbf)

shankari · 2023-07-18T18:18:52Z

our integration tests are currently in emission/integrationTests/
you should be able to use them as an example
we should initially configure to use the standard OSM-based nominatim, but once we get the geofabrik API key, we can also configure to use it with a github actions secret configuration.

Let's first write the test and then see whether we want to use the secrets or not. we might want to run the tests against both geofabrik and OSM nominatim initially to see if they are consistent . If they stay consistent for a while, we could consider removing one.

shankari · 2023-07-18T18:23:02Z

Right now, the URL for the nominatim service is in a config file (conf/net/ext_service/nominatim.json.sample). This is going to make it tricky to configure through secrets since the secrets are made available as environment variables in the github action. Since we are rewriting this code anyway, we might want to switch to reading from an environment variable anyway.

We could also configure the file using an environment variable, but I think that doesn't make as much sense given that:

there is only one entry in the file and
we are switching to docker for almost everything, and
docker works better with environment variables anyway

nataliejschultz · 2023-07-20T18:30:34Z

I decided to test the functionality of the docker containter that I downloaded. I started the docker container and modified the nominatim.json with this line:
"query_url" : "http://localhost:8080/nominatim".
In the REPL:

>>>reverse_geocoded_json = eco.Geocoder.get_json_reverse(43.729325, 7.416883) 
http://nominatim.openstreetmap.org/reverse?lat=43.729325&lon=7.416883&format=json 
>>> print(reverse_geocoded_json)
{'place_id': 64027309, 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright', 'osm_type': 'node', 'osm_id': 5918204054, 'lat': '43.7287949', 'lon': '7.417052', 'display_name': 'NOEUD 1987, Galerie Princesse Stéphanie, Fontvieille, Monaco, 98020, Monaco', 'address': {'tourism': 'NOEUD 1987', 'road': 'Galerie Princesse Stéphanie', 'suburb': 'Fontvieille', 'ISO3166-2-lvl10': 'MC-FO', 'city': 'Monaco', 'postcode': '98020', 'country': 'Monaco', 'country_code': 'mc'}, 'boundingbox': ['43.7287449', '43.7288449', '7.417002', '7.417102']}

I used a latitude and longitude in Monaco. I also modified nominatim.py with print(NOMINATIM_QUERY_URL ) to see how the url was being formatted.

The output was as expected. So, if the correct dataset is downloaded, the docker container method with nominatim could be useful.

nataliejschultz · 2023-08-02T18:48:49Z

Looked heavily into nominatim's documentation to see if we can query data from previous dates. Unfortunately, this is not possible for any call in nominatim.

As an alternative solution, tried to use overpy to query for the previous display name data and it did not work. It is possible to query past data using Overpass API by setting a date variable at the top:

[date:"2016-01-01T00:00:00Z"]

Resulting outputs were very large, and require a large amount of resources just to potentially get the display name out. You cannot query against addresses, unless they are very specific, which would require a large amount of looping over data. Additionally, you cannot query against lat/long, as it wants an input of bbox. Bbox outputs from our cleaned data were conflicting with requirements from the api.

shankari · 2023-08-02T19:14:38Z

we should file an issue/feature request about this with the nominatim team. Note that Nominatim is open source
https://github.com/osm-search/Nominatim
I don't think we need to explore the overpy approach for production in general because we will query ~ 1 hour after the trip is done so we will not get bad data
the real need for historical queries is:
a. in case we reset the pipeline and re-run it many months after the fact
b. testing

For 3 (a), we could actually use the overpy approach as a fallback if the location that we are lookup was older than a week or something.

For 3 (b) it doesn't make sense to only test the overpy to get the value if we are going to continue using nominatim as the default. If we are going to use nomimatim, we should test nominatim. So for testing alone, I would suggest that we:

create a set of integration tests that are run on GitHub actions once a week
have this integration test create a nominatim container with a particular known OSM dataset (so not "latest")
run several of the real world use cases against the local nominatim container and verify the results
we can pick a location and only include tests that are in that location in the integration test - probably CA since most of the test data is from California

For docker:

get familiar with the nominatim docker some more, experiment with it for loading Colorado and California data etc
understand test_with_docker because it launches a dockerfile within a GitHub action and we will now need to do this too
phone code has examples of running a GitHub action once a week

For the overpy approach, I don't see what the issue is.

it is not an issue that it can't look up by address, because we are trying to do lat,lon -> address
for the second approach, we have a lat/lon, we can construct a bbox by essentially adding and subtracting small numbers from it. As I said before, we already integrate with the overpass API in the pipelne to find the bus stops near the start and end of a trip or section. We can re-use (and potentially extend) that code to do what we want.
https://github.com/e-mission/e-mission-server/blob/83dad9b22458325ccd50a6c857fbd3c85c902b3e/emission/net/ext_service/transit_matching/match_stops.py#L115

If we implement the overpass python backup option, we should test that as well.

shankari · 2023-08-04T22:16:45Z

to run integration tests, you would:

make a new docker-compose.yml file that includes the database (db) and a nominatim service (nominatim), launched from the nominatim docker, and loading a particular snapshot of the OSM data and a webserver service for running the integration tests.

So similar to the existing docker-compose but with the addition of the nominatim service.

Couple of caveats to note are:

the webserver currently launches all tests (look at tests/Dockerfile); you will need to launch the integration tests instead. This will likely involve using https://docs.docker.com/compose/compose-file/compose-file-v3/#command to override the default command
you will likely need to use a similar approach to load data into nominatim
you will probably need to configure the webserver to use http://nominatim as the URL to use. I think we changed this to use an environment variable, so you would pass that in to using the env in docker-compose

nataliejschultz · 2023-08-07T17:21:29Z

During our meeting, we wanted to test that the original docker-compose command was working locally, so that the very basics of setting up a new container would not be an issue. While running
docker-compose -f setup/docker-compose.tests.yml up in terminal, we ran into the error ./runAllTests.sh: line 9: python: command not found. Originally, we thought this was an issue of the files not copying correctly. However, the key is that it is the python command that was not found, not the file.

Line 9 ofrunAllTests.sh reads as follows:

PYTHONPATH=. python -m unittest discover -s emission/tests -p Test*;

This resulted in a python command not found error, because my machine uses python3. Changing the command to PYTHONPATH=. python3 ... fixed the error and allowed docker-compose to succeed. I may be able to fix future python incompatibilities by setting alias python='python3' in my .bash_profile.

nataliejschultz · 2023-08-08T18:36:10Z

Locally, running the following block of code successfully downloads the OSM data from Northern California and creates a container named norcal-nominatim:

docker run -it \
  -e PBF_URL=https://download.geofabrik.de/north-america/us/california/norcal-latest.osm.pbf \
  -e REPLICATION_URL=https://download.geofabrik.de/north-america/us/california/norcal-updates/ \
  -e IMPORT_WIKIPEDIA=false \
  -v nominatim-data:/var/lib/postgresql/14/main \
  -p 8080:8080 \
  --name norcal-nominatim \
  mediagis/nominatim:4.2

It also creates a volume, so that the container can be started and stopped while maintaining the data.

This command line call is great for local testing, but our current digital server uses docker-compose to test. So, I converted the commands passed into docker run -it into commands for the docker-compose file:

  nominatim:
    image: mediagis/nominatim:4.2
    container_name: norcal-nominatim
    environment:
      - PBF_URL=https://download.geofabrik.de/north-america/us/california/norcal-latest.osm.pbf
      - REPLICATION_URL=https://download.geofabrik.de/north-america/us/california/norcal-updates/
      - IMPORT_WIKIPEDIA=false
    ports:
      - "8080:8080"
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    volumes:
      - nominatim-data:/var/lib/postgresql/14/main
    networks:
      - emission

This way, we can integrate running the nominatim server in tandem with our mongoDB, so that we can make calls to both at the same time. This might be necessary, depending on if the other integration tests will require calls to mongo.

To run the tests, I added a directory to the integration tests called nominatimTests. I created a file called nominatim-docker-test.yml, that runs the test every Sunday at 4:05am. Most notably, it builds a Dockerfile in the integrationTests directory:

web-server:
    build: ../../integrationTests

and runs docker-compose:

name: Initialize nominatim 
run: docker-compose -f docker-compose.yml up --exit-code-from web-server

The Dockerfile runs a file called start_integration_tests.sh, which will initialize all of the integration tests.

nataliejschultz · 2023-08-08T18:47:52Z

On my test run, the servers were initializing (though seemed to be dueling due to using the same port). I tried to query the local nomination server, and it did not work. I decided to do a quick sanity check and make sure that my goal was possible by following the procedure on the nomainatim-docker page for Monaco. I ran their docker run -it and called the server using their suggested command:
curl -v http://localhost:8080/search.php?q=avenue%20pasteur
and received a response successfully.

I decided to try a local test using the Northern California container, but modify the common by including the optional wikipedia import, while removing the creation of the volume. This way, it would be more similar to the command used in the successful Monaco example. This increased the time required to complete the docker run -it exponentially (almost 45 minutes!), but will hopefully allow for a reverse geocoding call to succeed.

shankari · 2023-08-08T19:06:07Z

Line 9 ofrunAllTests.sh reads as follows:

PYTHONPATH=. python -m unittest discover -s emission/tests -p Test*;

This resulted in a python command not found error, because my machine uses python3. Changing the command to PYTHONPATH=. python3 ... fixed the error and allowed docker-compose to succeed. I may be able to fix future python incompatibilities by setting alias python='python3' in my .bash_profile.

This makes no sense. The whole point of running in docker is that it is isolated from what is on your machine. Even on your machine, if you use the version of python from the e-mission environment, you should not have to use python3, regular python should work

shankari · 2023-08-08T19:27:58Z

So we tried to run the container again using
e-mission-server$ docker-compose -f setup/docker-compose-tests.yml up

It did appear to launch the tests, although we don't fully know why

We execed into the container with docker exec -it <container> /bin/bash and found that there was nothing in /usr/src/app

# cd /usr/src
# ls

where is this code running from then? Are we mounting the source code of the test somewhere?

mounting happens in the docker-compose file
find / -name \*emission\* -print to check the container filesystem as well

nataliejschultz · 2023-08-08T21:44:33Z

Line 9 ofrunAllTests.sh reads as follows:

PYTHONPATH=. python -m unittest discover -s emission/tests -p Test*;

We tested out the docker-compose again and found that writing python3, while resolving the python command not found error initially, did not entirely resolve the problem.

We execed into the container with docker exec -it /bin/bash and found that >there was nothing in /usr/src/app

The server is in src/e-mission-server. Once in that directory, running source setup/activate.sh produced the error:

Could not find conda environment: emission

Running source setup/activate_conda.sh still did not allow the e-mission environment to run, so the python command was still not able to be run. Terminal suggests running conda info --envs to display the available environments, which produces these options:

base                     /root/miniconda-23.1.0
emissiontest             /root/miniconda-23.1.0/envs/emissiontest

shankari · 2023-08-08T22:00:22Z

@nataliejschultz Right, you need to activate emissiontest. again, try to run the start script commands one by one in the container and see what you get. Concretely, note

echo "Setting up the test environment..."
source setup/setup_tests.sh

echo "Running tests..."
source setup/activate_tests.sh

before running all tests

shankari · 2023-08-09T02:04:37Z

@nataliejschultz your issues with the local docker-compose are almost certainly a configuration issue on your side. As you can see, the CI test is running just fine, and I can verify that the tests are launched properly when I try to run.

$ docker-compose -f setup/docker-compose.tests.yml up
...
setup-web-server-1  | Found match for {'_id': ObjectId('64d2e0a02dec3b0cfd6065a7'), 'track_location': {'type': 'Point', 'coordinates': [-122.259169, 37.873873]}}
setup-web-server-1  | Found match for {'_id': ObjectId('64d2e0a02dec3b0cfd6065a8'), 'track_location': {'type': 'Point', 'coordinates': [-122.25874, 37.875711]}}
setup-web-server-1  | Found match for {'_id': ObjectId('64d2e0a02dec3b0cfd6065a9'), 'track_location': {'type': 'Point', 'coordinates': [-122.254577, 37.870352]}}
setup-web-server-1  | Setting up real example for 3c67379e-a214-402a-ab11-ea56ba822b97
setup-web-server-1  | Setting up real example for dc2bd19d-7ab5-4939-877e-a40ab384f6e9
setup-web-server-1  | Setting up real example for fbe2562c-d318-4741-985c-f8e93a5b2469
setup-web-server-1  | Setting up real example for 594f9d91-7245-4b16-a44c-a887887a8f9b
setup-web-server-1  | Setting up real example for 84fc72d5-ea49-49ff-8c7a-6fccee89ac21
setup-web-server-1  | Setting up real example for 775799e4-b6b7-4974-8ca2-f731d8d88a34
setup-web-server-1  | Setting up real example for d2eb9e84-44f9-43aa-8fbc-0d00f88cec3c
setup-web-server-1  | Setting up real example for 63efc7ab-1b20-4faa-bbf8-ac47364ba8d1
setup-web-server-1  | Setting up real example for 498f37ad-05c0-4d2f-80d7-1b0cc8dd9a6f
setup-web-server-1  | Setting up real example for 86edfbef-50ee-4659-8488-5b87a8eb03ce
setup-web-server-1  | Setting up real example for ca934fb5-1b3e-4bb9-a382-9d88762f6092
setup-web-server-1  | Setting up real example for 6fef7b65-6822-4faa-bf2c-4cc5f39ecb6b
setup-web-server-1  | Setting up real example for 9682c622-9ced-441d-b3f1-f97acabdec47
setup-web-server-1  | Setting up real example for e4d6af87-d5c9-4cda-becb-57f639230c3b
setup-web-server-1  | Setting up real example for 232bdddb-5f6e-484c-bf54-77036233d505
setup-web-server-1  | Setting up real example for bfe873f8-1b03-45a7-8725-8a179915182c
setup-web-server-1  | Setting up real example for 3a4b3bb8-7d2b-4d20-8f65-7bd82a0cf7c0
setup-web-server-1  | Setting up real example for 3c05b16d-12e4-4042-a5b7-bf0641da7b85
setup-web-server-1  | Setting up real example for dd808145-0e39-4c13-9b1b-1e45600cec7d

I would encourage you to clear your environment and re-test, following the instructions in the CI until you get this to work successfully.

nataliejschultz · 2023-08-09T15:57:48Z

I've found that docker desktop has a helpful interface for looking through the filesystem of a container:

nataliejschultz · 2023-08-09T17:05:02Z

I tried a few different fixes, including running in my emission environment, but nothing was working to resolve the error python: command not found when trying to run setup_tests.sh and setup_conda.sh. However, I noticed a message that kept showing up right before the error:

setup-web-server-1  | #
setup-web-server-1  | # To activate this environment, use
setup-web-server-1  | #
setup-web-server-1  | #     $ conda activate emissiontest
setup-web-server-1  | #
setup-web-server-1  | # To deactivate an active environment, use
setup-web-server-1  | #
setup-web-server-1  | #     $ conda deactivate

So, I added a single line:
conda activate emissiontest
to setup_tests.sh, right before it calls python. This resolved the issue locally.

I noticed that the same single line of code is present in activate_tests.sh, which is executed after both setup_tests.sh and setup_conda.sh. Is there a reason that the environments are activated in the order that they are? Could activating the environment earlier cause problems down the line, or potentially be beneficial to avoid future configuration issues?

nataliejschultz · 2023-08-09T19:09:40Z

Have spent the past hour trying to figure out why my reverse geocoding call wasn't working locally, but would work using nominatim directly. I was using
curl -v http://localhost:8080/reverse?format=json&lat=37.8019701&lon=-122.419601,

and getting

{"error":{"code":400,"message":"Need coordinates or OSM object to lookup."}}.
However, when calling in the same style exemplified in nominatim-docker's repo:

curl -v http://localhost:8080/search?q=street%20lombard
the call succeeds. This verified that the issue was with the format of the reverse geocoding command. After changing things around with little success, I dug into the nomainatim-docker repo and found this page where someone was getting the same error.

Turns out that you need to put quotes around a call if it contains an ampersand while using curl. Finally succeeded with this:

curl -v "http://localhost:8080/reverse?format=json&lat=37.8019701&lon=-122.419601"

Now wondering if using postman would be a good option, if it's allowed?

nataliejschultz · 2023-08-09T20:25:34Z

Currently working on creating a fake trip in Rhode Island since I think the Northern California data will take too long to copy every time we test. Using the docker container method described in synthpop.

shankari · 2023-08-09T20:33:30Z

Now wondering if using postman would be a good option, if it's allowed?

I don't understand this comment. Where would we need to use postman?
I can understand using curl for testing, but presumably we are using python code for reverse geocoding in production and that is what we are testing?

shankari · 2023-08-09T20:36:38Z

Using the docker container method described in synthpop.

Why do we need to use synthpop? We don't need to simulate a synthetic population. And there is no US Census data for Monaco anyway. Just pass in lat/lon points known to be in Monaco/Rhode Island/...

Note previous comment:

for an example of how to create a fake trip (you can use a similar method to create a fake place): emission/tests//analysisTests/userInputTests/TestConfirmedObjectFakeData.py or similar

shankari · 2023-08-09T20:49:36Z

This resolved the issue locally.

You should not have to make any code changes to resolve locally. If you do, you are not running the local system correctly.
The CI/CD is running properly and so is my local copy.

noticed that the same single line of code is present in activate_tests.sh, which is executed after both setup_tests.sh and setup_conda.sh. Is there a reason that the environments are activated in the order that they are?

Because we need to setup before we activate. And once we have setup, we don't need to activate over and over.
it is the same pattern that we use for setting up the non test environments.

You need to see where the python error is coming from. Again, I would suggest running the startup script in the docker container step by step so you can see where the error is coming from.

Think of unix/HPC. A program that runs locally but not on the remote server is not useful.

nataliejschultz · 2023-08-11T16:57:41Z

After our meeting yesterday, we found out that the python: command not found error that was arising did not impact the tests passing. Might be something to address later on, but it's not important at the moment for our purposes.

Currently looking at how to address the comments on my commits last night in the PR for this issue.

nataliejschultz · 2023-08-15T16:37:19Z

Haven't been logging progress very much as I was feeling very sick the past few days. Have been steadily working on addressing the comments, as well as issues I've encountered along the way. I spent time trying to understand how to get the environment variable for NOMINATIM_QUERY_URL to change in the testing environment, rather than changing the nominatim.py file. After finding out about "monkey patching", my view on variables was changed! I modified all of the query variables in the test files to eco.NOMINIATM_QUERY_URL, and this fixed my issue. Now, the query URL can be temporarily changed to the localhost:8080 URL required to run nominatim in the docker container during testing, but keep nominatim.json the same.

nataliejschultz · 2023-08-15T17:50:05Z

Testing locally from docker-compose.yml. I also modified the workflow file nominatim-docker-test.yml to run on push to see how far it would get into the workflow. After looking into the error:

ERROR: for nominatim  Cannot create container for service nominatim: No command specified

I ran docker image inspect mediagis/nominatim locally. Found that "Cmd": null. Not sure what to set the command to at this point, but looking into it.

nataliejschultz · 2023-08-28T23:06:45Z

I had to add to the timeout a couple of times, but the Connection refused error was resolved once I set -timeout 240s. I am not sure if this is better than sleep, since the method doesn't actually wait for nominatim to initialize, and ends up using a timer anyway.

Despite running properly with dockerize, the tests are still failing. I noticed that my method to get the place_id to be equal using unittest's mock.ANY is still showing as a diff in both test_get_json_reverse and test_get_json_geo:

web-server_1  | -  'place_id': <ANY>}
web-server_1  | +  'place_id': 135108}

However, this is apparently a known error that might not be fixable. I might just have to change the assertion to expect a specific key (or multiple keys associated with that function).

Additionally, test_get_json_geo appears to be returning an empty list:
AssertionError: Lists differ: [{'place_id': <ANY>, 'licence': 'Data © Op[413 chars]001}] != []
I had this issue locally a few times, and changed the query to be more specific to fix it. I don't know why it's happening on GitHub Actions and not locally, since my local tests are still passing.

nataliejschultz · 2023-08-30T16:58:48Z

use curl to talk to the web API and use our python library to talk to the web api - is the library working correctly and is it doing so against the current version of the web api? is it maintaining forwards compat

This has proven to be an issue. I have been trying to use requests, which works in a regular python3 kernel to call the web API, which works as expected:

import requests
nominatim_url = "http://nominatim.openstreetmap.org/reverse?lat=41.8239891&lon=-71.4128343&format=json"
nominatim_result = requests.get(nominatim_url)
nominatim_result = nominatim_result.json()
print(nominatim_result)
{'place_id': 132765501, 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright', 'osm_type': 'way', 'osm_id': 121496393, 'lat': '41.824034499999996', 'lon': '-71.41290469687814', 'class': 'amenity', 'type': 'townhall', 'place_rank': 30, 'importance': 0.2257940944999783, 'addresstype': 'amenity', 'name': 'Providence City Hall', 'display_name': 'Providence City Hall, 25, Dorrance Street, Downtown, Providence, Providence County, Rhode Island, 02903, United States', 'address': {'amenity': 'Providence City Hall', 'house_number': '25', 'road': 'Dorrance Street', 'neighbourhood': 'Downtown', 'city': 'Providence', 'county': 'Providence County', 'state': 'Rhode Island', 'ISO3166-2-lvl4': 'US-RI', 'postcode': '02903', 'country': 'United States', 'country_code': 'us'}, 'boundingbox': ['41.8237547', '41.8243153', '-71.4132816', '-71.4125278']}

However, when using the emission kernel, or trying to call from within the environment inside the docker container, I get a 403 forbidden error.

shankari · 2023-08-30T17:05:14Z

However, this is apparently a pytest-dev/pytest#3638 that might not be fixable. I might just have to change the assertion to expect a specific key (or multiple keys associated with that function).

I would just delete the place_id from both results before comparing. Since most of the keys are the same, it will be easier and more robust to remove the key that is not working

shankari · 2023-08-30T17:12:06Z

However, when using the emission kernel, or trying to call from within the environment inside the docker container, I get a 403 forbidden error.

This is almost certainly due to our running afoul of nominatim's usage limits. We are not supposed to make too many calls.

Given that it works on your laptop in the regular python3 but not in emission, I would think that this may just be the random ordering chance.

you made some calls from python3
you used up the limit
new calls don't work

not really related to the python environment

Couple of things here:

this is good because it let's us see how we handle errors
do we actually log the error properly? How were you able to identify that this was a 403?

We will not actually use nominatim for this in production because that will definitely violate their terms. We should use the geofabrik API to test against instead.

Note that in that case, you cannot hardcode the URL since it needs to embed our API key which we do not want to publish. Fortunately, we should already be using an environment variable to pass in the URL to call.

So you can use an ennvironment variable in docker-compose for local testing and on GitHub actions, use GitHub actions secrets to pass in the variable with the API key embedded.

shankari · 2023-08-30T17:32:21Z

The module we are trying to test in the pipeline only works on a place. It does not need a trip. If you run it and it fails because it doesn't have a trip, we have to think about it but we can keep that around as a negative test.

But I don't think it needs a trip (90% certainty).

So you create a fake place, and pass it in!

The main difference between this and calling the module directly is that the module will return the result to you directly, but the pipeline stage will fill in the result into the place and return the place.

So calling the pipeline stage is testing that the stage reads the lat/lon from the place and passes it to the module correctly and extracts the address correctly and fills in the response correctly.

You should test with a blank data place, but expect it to fail
you should then test with a place with lat/lon filled in per the data model and expect it to pass

nataliejschultz · 2023-09-11T16:51:17Z

I am back from my trip! I spent the first chunk of the day dealing with timesheet issues (very unfortunate timing with the end of the fiscal year), but am working on creating a fake place now.

nataliejschultz · 2023-09-12T18:49:40Z

I created a fake place using the create_fake_entry function in Entry.py. I input the following:

id = "rhodeislander"
key = "segmentation/raw_place"
write_ts = 1
data = {'source': 'FakeTripGenerator','enter_local_dt': {'hour': 11, 'month': 9,'second': 13,'weekday': 6,'year': 2023,'timezone': 'America/New_York','day': 10,'minute': 12}, 'exit_local_dt': {'hour': 11, 'month': 9,'second': 59,'weekday': 6,'year': 2023,'timezone': 'America/New_York','day': 10,'minute': 12},'location': {'type': 'Point', 'coordinates': [-71.4128343, 41.8239891]}, 'duration': 46.77990412712097}
fake_place = ecwe.Entry.create_fake_entry(id, key, data, write_ts)

And it outputs an Entry class that looks like this:

Entry({'_id': ObjectId('6500b09f949a2ec1b9e84712'), 'user_id': 'rhodeislander', 'metadata': Metadata({'key': 'segmentation/raw_place', 'platform': 'server', 'write_ts': 1, 'time_zone': 'America/Los_Angeles', 'write_local_dt': LocalDate({'year': 1969, 'month': 12, 'day': 31, 'hour': 16, 'minute': 0, 'second': 1, 'weekday': 2, 'timezone': 'America/Los_Angeles'}), 'write_fmt_time': '1969-12-31T16:00:01-08:00'}), 'data': {'source': 'FakeTripGenerator', 'enter_local_dt': {'hour': 11, 'month': 9, 'second': 13, 'weekday': 6, 'year': 2023, 'timezone': 'America/New_York', 'day': 10, 'minute': 12}, 'exit_local_dt': {'hour': 11, 'month': 9, 'second': 59, 'weekday': 6, 'year': 2023, 'timezone': 'America/New_York', 'day': 10, 'minute': 12}, 'location': {'type': 'Point', 'coordinates': [-71.4128343, 41.8239891]}, 'duration': 46.77990412712097}})

The function fills in the metadata per instructions in metadata.py's create_metadata_for_fake_result function, which does not match the data that I provided. However, the metadata does not affect reverse geocoding when I pass my fake_place class into the get_filtered place function, since it only needs the lat and long from the location key:

 x = clean.get_filtered_place(fake_place)
>>>reverse_geocoded_json: {'place_id': 16819435, 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright', 'osm_type': 'way', 'osm_id': 121496393, 'lat': '41.824034499999996', 'lon': '-71.41290469687814', 'class': 'amenity', 'type': 'townhall', 'place_rank': 30, 'importance': 0.2257940944999783, 'addresstype': 'amenity', 'name': 'Providence City Hall', 'display_name': 'Providence City Hall, 25, Dorrance Street, Downtown, Providence, Providence County, Rhode Island, 02903, United States', 'address': {'amenity': 'Providence City Hall', 'house_number': '25', 'road': 'Dorrance Street', 'neighbourhood': 'Downtown', 'city': 'Providence', 'county': 'Providence County', 'state': 'Rhode Island', 'ISO3166-2-lvl4': 'US-RI', 'postcode': '02903', 'country': 'United States', 'country_code': 'us'}, 'boundingbox': ['41.8237547', '41.8243153', '-71.4132816', '-71.4125278']}

The reverse geocoding is successful!

shankari · 2023-09-12T19:11:01Z

I don't think you need enter_dt and exit_dt since we don't care about the timestamps for this call (yet).
And even if we did, we would use the timesetamp instead of the local_dt

data = {'source': 'FakeTripGenerator','enter_local_dt': {'hour': 11, 'month': 9,'second': 13,'weekday': 6,'year': 2023,'timezone': 'America/New_York','day': 10,'minute': 12}, 'exit_local_dt': {'hour': 11, 'month': 9,'second': 59,'weekday': 6,'year': 2023,'timezone': 'America/New_York','day': 10,'minute': 12},'location': {'type': 'Point', 'coordinates': [-71.4128343, 41.8239891]}, 'duration': 46.77990412712097}

one of the issues you have been facing is that it takes a long time to start up the nominatim docker container. I know you have worked around it with timeouts but we still don't want to abuse the github actions by making it do a lot of work. Since you are creating the container with the same data load every time, you can use docker commit to create an image with the data pre-loaded. You would then push that up to dockerhub and use it directly as the base image while starting up the container. You would still need to wait for the container to start up, but it should be a lot faster and less resource intensive than loading the data and building the database.

We can simply update the image every time we update the data file, roughly once a year.

shankari · 2023-09-12T19:14:46Z

one test against the base nominatim API for community users who don't have a paid subscription (this should not trigger usage limits). Other tests against the geofabrik API which is the one we use, should be compatible with nominatim, and should not have usage limits. Depending on test stability we can decide how to proceed.

nataliejschultz · 2023-09-14T07:57:37Z

I have been working on this for a very long time now, and have a lot of thoughts to share. I really wanted to get it done, but there are too many issues to deal with at the moment. The most important test is passing, as are the majority of the other 9 tests. I'm not supposed to work tomorrow according to my schedule, but I am very determined and want to get this done with so I can move on!! I'll get online in the morning when I can collect my thoughts properly.

nataliejschultz · 2023-09-19T17:35:46Z

Here is the long update (pt 1):

I removed the enter_dt and exit_dt from my fake place as @shankari suggested, and it still worked fine when being passed into get_filtered_place.

Workflow Woes:
An issue arose where the server wasn't fully initializing. Because of this, the workflow would sit and do nothing until it got kicked off the runner. I had accidentally uncommented out a tail -f /dev/null command in one of the files that starts up the server, and re-commenting it out fixed it. In hindsight, I can see how this could cause problems. Trying to fix this issue led me to figuring out how to move the job to my forked branch, which gave me control of the workflow. I can now cancel workflows if they get stuck, and have been testing with API keys.

API Key Woes:
I have tried a few different ways to call geofabrik with the secret that I added to GitHub Actions Secrets on my fork. It is currently unclear to me if it is possible to set the geofabrik url with the secret embedded as an environment variable to be used within the program, or if it can only be used within the nominatim-docker-test.yml file. Note that in the following , I am referring to runs on my forked branch, where I have added the real secret and a fake secret for testing to GitHub Actions Secrets.

I tried adding a few different environment variables to the test in nominatim-docker-test.yml:

- name: Test nominatim
      env:
        GEOFABRIK_QUERY_URL: "https://geocoding.geofabrik.de/${{secrets.GEOFABRIK_API}}"
        TEST_KEY: ${{secrets.TEST_SECRET}}
        TEST_ENVVAR: "https://geocoding.geofabrik.de/${{secrets.TEST_SECRET}}"
        TEST_STR: "https://geocoding.geofabrik.de/"

Then, I attempted two different methods to access the secrets in my TestNominatim.py file:

TEST_KEY = os.environ["TEST_KEY"]
TEST_2KEY = os.environ.get("TEST_KEY")

The first method resulted in an error:

File "/src/e-mission-server/emission/individual_tests/TestNominatim.py", line 27, in <module>

web-server_1  |     TEST_KEY = os.environ["TEST_KEY"]

…

web-server_1  | KeyError: 'TEST_KEY'

So I continued with the second method. It seems like the variables are not carried through to this file anyway, since my prints:

TEST_2ENVVAR = os.environ.get("TEST_ENVVAR")
print("get method", TEST_2ENVVAR)
TEST_2KEY = os.environ.get("TEST_KEY")
print("get method key", TEST_2KEY)
TEST_STR = os.environ.get("TEST_STR")
print("TESTSTRING", TEST_STR)

Yield the following result:

web-server_1  | get method None

web-server_1  | get method key None

web-server_1  | TESTSTRING None

Additionally, I have not found any examples of other people using these supposed environment variables outside of their yml files. I think at this point I will have to use AWS secrets manager.

nataliejschultz · 2023-09-19T18:13:17Z

Tests
Currently, one test is failing and the other two are erroring out.

test_geofabrik_and_nominatim is erroring because I cannot use the API keys. This will be a straightforward fix.

test_nominatim_api is failing because it includes a hard-coded nominatim call. Nominatim released a new version on the 7th. The current container that we are comparing with returns a slightly different result. I am not sure when the 4.3.0 container will be ready.

test_get_json_geo is erroring because it is returning an empty list. Here is the test:

    def test_get_json_geo(self):
        expected_result = "Hartford Pike"
        actual_result = eco.Geocoder.get_json_geo("Old Hartford Pike, Scituate, RI 02857")[0]["name"]
        self.assertEqual(expected_result, actual_result)

It calls get_json_geo, which is supposed to pass in an address and return a list of a dictionary with nominatim data on that location. It might be failing because I skipped adding the wikipedia data to the container. However, I don't know if this test is completely necessary.

The most important test, which is test_get_filtered_place, passes in my fake place to get_filtered_place:

def test_get_filtered_place(self):
        raw_result = ecww.WrapperBase.__getattr__(clean.get_filtered_place(fake_place), "data")
        print(NOMINATIM_QUERY_URL)
        actual_result = ecww.WrapperBase.__getattr__(raw_result, "display_name")
        expected_result = "Dorrance Street, Providence"

This test is passing.

Thoughts

Originally, we used the container because we couldn't call nominatim from a specific date. However, since we can directly call both geofabrik and nominatim in the web server, I'm not sure if using the container is entirely necessary.
The most important test, which tests get_filtered_place, is passing. I can easily remove the tests that aren't working yet so that the most important test can be implemented.

nataliejschultz · 2023-09-19T18:43:16Z

   def test_nominatim_api(self):
        nominatim_url = "http://nominatim.openstreetmap.org/reverse?lat=41.832942092439694&lon=-71.41558148857203&format=json"
        nominatim_result_raw = requests.get(nominatim_url)
        nominatim_result = nominatim_result_raw.json()['display_name']
        # NOMINATIM_QUERY_URL = eco.NOMINATIM_QUERY_URL
        docker_result = eco.Geocoder.reverse_geocode(41.832942092439694, -71.41558148857203)

shankari · 2023-09-19T18:50:34Z

we should get test_geofabrik_and_nominatim to work
as you can see, the nominatim API has already changed, which is why these tests are important
for test_nominatim_api, I don't see why we need a new container. This tests against against the current tip of nominatim.
We have the container based test so that if the container test passes, but test_nominatim_api fails, we know that it is due to a nominatim update and can potentially change our API call. That is (or should be) exactly the case that we are in now (assuming you had the container working properly). I don't see why not having the container will fix that
- Have you used docker commit or are you still loading the container from scratch every time?
given that the nominatim API has changed, and test_nominatim_api fails, I don't see how test_get_filtered_place passes. Why isn't the "slightly different result" putting this off? It seems like either test_nominatim_api is too strict or test_get_filtered_place is not correct.

nataliejschultz · 2023-09-20T04:04:32Z

Big Progress!

I tried many different ways to pass the test API key through to TestNominatim.py. I eventually found out that I have to pass the variable into the docker-compose through the docker-compose command, which can be done like this:

run: TEST_KEY=${{ secrets.TEST_SECRET }} docker-compose -f emission/integrationTests/docker-compose.yml up --exit-code-from web-server

Then, the variable has to be set again in the docker-compose (specifically in the web-server service in the environment section) so that it can be accessed by the web server when running the test:

web-server:
 …
    environment:
      - TEST_KEY=$TEST_KEY

I wanted to make sure that the secret was still being recognized as a secret, and not just a regular k/v. So, I added a print to TestNominatim.py:

print("keys", os.environ.keys())

Which yields the following in GitHub Actions:

keys KeysView(environ({… ’TEST_KEY': '***' …}))

So it appears that the secret is still protected, even after passing through multiple files. I am going to test with the real key and see if I can get that test to work!

shankari · 2023-09-20T14:43:15Z

Big Progress!

Indeed!

I tried many different ways to pass the test API key through to TestNominatim.py. I eventually found out that I have to pass the variable into the docker-compose through the docker-compose command, which can be done like this:

Great job digging through this and figuring out how these environment variables are passed through in GH actions. You should put that into your demo for others to learn!

nataliejschultz · 2023-09-21T17:40:04Z

Currently, test_geofabrik_and_nominatim is passing!

test_get_json_geo is passing locally, but giving this error on actions:
urllib.error.HTTPError: HTTP Error 500: Internal Server Error`

This is strange, because the same function is called earlier in the program and not producing this error.

test_nominatim_api, which compares our container result to a direct nominatim query, is not passing because of nominatim switching versions and changing the results. With their new version, nominatim outputs the name of the state (Rhode Island) in the result for the display name. This is different from how the container outputs the display name:

 AssertionError: 'Stat[77 chars]vidence, Providence County, Rhode Island, 02908, United States' != 'Stat[77 chars]vidence, Providence County, 02908, United States'
web-server_1  | - State of Rhode Island Department of Administration, 1, Park Street, Downtown, Providence, Providence County, Rhode Island, 02908, United States
web-server_1  | ?                                                                                                              --------------
web-server_1  | + State of Rhode Island Department of Administration, 1, Park Street, Downtown, Providence, Providence County, 02908, United States

I am glad that this showed up, because it means that the test is working in the way that we had hoped by identifying a change in the way that nominatim displays outputs. I'm not sure if I should modify the test to ignore this part of the result since we are now aware of the change.

nataliejschultz · 2023-09-21T18:23:29Z

It looks like the HTTP error was temporary, because the tests are finally PASSING!!!

Moving the code to ready for review!

nataliejschultz · 2023-09-26T05:41:28Z

Since the tests were passing, I decided to do the docker commit work. I created a docker hub account, built my container locally by running docker compose, created an image from the container using docker commit, and pushed that image to docker hub. I then modified the workflow file to pull my newly created image.
After running locally, it worked exactly the same as it had before!

When I looked through the logs, actions appears to be going through the exact same process as before; essentially, it's still building the container from scratch:

Pulling nominatim (nataliejschultz/rhodeisland-image:2.0)...
Status: Downloaded newer image for nataliejschultz/rhodeisland-image:2.0
...
rhodeisland-nominatim | + '[' https://download.geofabrik.de/north-america/us/rhode-island-latest.osm.pbf '!=' '' ']'
rhodeisland-nominatim | + echo Downloading OSM extract from https://download.geofabrik.de/north-america/us/rhode-island-latest.osm.pbf
rhodeisland-nominatim | + curl -L -A mediagis/nominatim-docker:4.2.3 --fail-with-body https://download.geofabrik.de/north-america/us/rhode-island-latest.osm.pbf -C - --create-dirs -o /nominatim/data.osm.pbf
rhodeisland-nominatim | Skipping optional Wikipedia importance import
rhodeisland-nominatim | Skipping optional GB postcode import
rhodeisland-nominatim | Skipping optional US postcode import
rhodeisland-nominatim | Skipping optional Tiger addresses import

It seems like pre-building the container and committing it as an image didn't make any difference. Despite still passing, the test took a minute longer.

shankari · 2023-09-26T17:26:56Z

@nataliejschultz I don't see the tests passing in the PR. Can you add a link to the tests passing on your fork? I can then know that this is because I haven't configured the secret yet.

It seems like pre-building the container and committing it as an image didn't make any difference. Despite still passing, the test took a minute longer.

This is weird. Can you confirm that when you "built my container locally by running docker compose", you waited for the container to start up before using git commit? If you did, you may want to look into the endpoint for the container.

This may be because the endpoint downloads and builds without checking to see if the data has already been loaded. In that case, you should override the endpoint in the workflow file to only include the final "run" step (since the download + install is encapsulated in your committed image).

nataliejschultz · 2023-09-26T18:03:55Z

Can you add a link to the tests passing on your fork? I can then know that this is because I haven't configured the secret yet.

Here is a link to a successful run.
The other tests that run on pull request are still running on the main server branch every time I push (since all my pushes are currently linked to PR #929), but my nominatim test is running exclusively on my fork.

Can you confirm that when you "built my container locally by running docker compose", you waited for the container to start up before using git commit? If you did, you may want to look into the endpoint for the container.

I waited for docker compose to finish completely (ie for the tests to run locally), exited the container, and ran these commands in terminal:

docker commit rhodeisland-nominatim nataliejschultz/rhodeisland-image:2.0
docker push nataliejschultz/rhodeisland-image:2.0

I just tried creating the image off of a running container and pushing that (see image tag 3.0), and it is the exact same size as the 2.0 image.

Here is what it says in my terminal when I created and pushed the images:

nschultz$ docker commit rhodeisland-nominatim nataliejschultz/rhodeisland-image:3.0
sha256:d14a75faad26c3c1c10bad125ec67ac5c48b7cde428a8d01d61157e70a5387fe
nschultz$ docker push nataliejschultz/rhodeisland-image:3.0
The push refers to repository [docker.io/nataliejschultz/rhodeisland-image]
6f91b3895bca: Pushed 
682fe9f7f417: Layer already exists 
3da28f0d74bf: Layer already exists 
5f70bf18a086: Layer already exists 
1bd0eca487ba: Layer already exists 
3.0: digest: sha256:5f712008842ec27816ed394d21ad40f4bca4228f8d380488e72e236876d1e974 size: 1361

This may be because the endpoint downloads and builds without checking to see if the data has already been loaded. In that case, you should override the endpoint in the workflow file to only include the final "run" step (since the download + install is encapsulated in your committed image).

Once I created my new image, I modified the docker compose for the nominatim service to look like this:

nominatim:
    image: nataliejschultz/rhodeisland-image:2.0
    container_name: rhodeisland-nominatim
    ports:
      - "8080:8080"
    healthcheck:
      test: ["CMD", "curl", "-v", "http://localhost:8080"]
      interval: 1m
      timeout: 15s
      retries: 3
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    volumes:
      - nominatim-data:/var/lib/postgresql/14/main
    networks:
      - nominatim

The workflow file itself only runs the compose; are you suggesting I should modify the workflow file to run the image and start a container (if possible), or modify the nominatim service in the compose?

shankari · 2023-09-26T18:06:47Z

modify the nominatim service in the compose. I used an admittedly imprecise description to try and align to what you had said before "I then modified the workflow file to pull my newly created image." in an attempt to make it easier to understand 😄 Will make sure to stay precise in the future!

nataliejschultz · 2023-09-26T18:19:26Z

Will make sure to stay precise in the future!

As should I! My bad.

shankari · 2023-09-26T18:55:40Z

start.sh calls init.sh if the IMPORT_FINISHED file doesn't exist
you might want to first check to see if IMPORT_FINISHED is in the committed image by using docker run.../bin/bash or docker exec ... /bin/bash

But it might just be that /var/lib/... is not committed as part of the snapshot.

I can think of three ways to fix this:

set IMPORT_FINISHED to a different file location. But I am not sure this will work since start.sh just sets IMPORT_FINISHED and doesn't seem to support any overrides
make sure that the file exists
- mount some file to /var/lib/..... under the volumes (bind mount) OR
- override the entrypoint to be touch /var/lib/... && /app/start.sh

nataliejschultz · 2023-09-27T17:34:40Z

I spent time rooting around in one of the images I created using
docker run -it nataliejschultz/rhodeisland-image:3.0 /bin/bash, and the import-finished file was not present. However, echo $PBF_URL gave a result, indicating that the configuration script had already been run.

I started by trying to run start.sh manually (just to see what would happen), and this got the resultant container too far into the process to be used in the test.

Starting over and working on a different image, I ran both config.sh and init.sh without running the start script. All of the Rhode Island files were downloaded. It also created a container, which I committed and pushed as an image to docker hub. This image is much larger (over a GB in size!) than the previous images.

Running my workflow outright in another test gave an error right away. I wasn't sure if this was going to happen, but I had my suspicions.

Now that I have an image to use that has the correct data loaded, I believe I can change the entrypoint and get it to run the way we want!

nataliejschultz · 2023-09-28T20:37:38Z

Okay, so it wasn't as simple as just changing the entrypoint. I have a suspicion that the file required to bypass init.sh is immediately destroyed upon using docker commit (which automatically shuts down the container!). So, I went into my container before using docker commit and used touch import_finished. I pushed this image to the hub and ran the tests. It finished in less than 3 minutes (!!!), though some of the other changes I made caused the tests to fail. Still, it runs almost twice as fast now, and has cut a lot of junk from the logs. I'm removing health check now, which should remove even more unnecessary lines from the logs. Addressing many of Shankari's comments along the way.

ps: planned to get more done yesterday, but my home had a leak we had to address :/ working today to compensate.

nataliejschultz · 2023-09-29T02:27:08Z

I've edited all of the tests as much as I can without further input from @shankari :) I'm really excited for this test to be done!!

nataliejschultz · 2023-10-02T21:38:50Z

Today has been super busy!! I met with Abby about the Samsung phone and went over my many trips of different types that I've taken over the past few weeks. Also had the all-hands, during which I couldn't test locally.

I've been addressing the comments in my PR, but haven't pushed changes yet.

nataliejschultz · 2023-10-09T23:13:16Z

Okay, I think the URL call issue is finally fixed. It took me a long time to figure out how to get it right. Here's an example of the current workflow:

Set environment variables for web-server service in docker-compose:

    environment:
      - DB_HOST=db
      - GFBK_KEY=$GFBK_KEY
      - GEOFABRIK_QUERY_URL=https://geocoding.geofabrik.de/$GFBK_KEY
      - OPENSTREETMAP_QUERY_URL=https://nominatim.openstreetmap.org
      - NOMINATIM_CONTAINER_URL=http://rhodeisland-nominatim:8080

TestNominatim.py gets these variables:

OPENSTREETMAP_QUERY_URL = os.environ.get("OPENSTREETMAP_QUERY_URL")
GEOFABRIK_QUERY_URL = os.environ.get("GEOFABRIK_QUERY_URL")
NOMINATIM_CONTAINER_URL = os.environ.get("NOMINATIM_CONTAINER_URL")

I left them this way because the string of the URL is used in a few different tests.

Nominatim module sets NOMINATIM_QUERY_URL to one of the URLs, based on how it's called:

 def nominatim(service):
        if service == "container":
            os.environ["NOMINATIM_QUERY_URL"] = NOMINATIM_CONTAINER_URL
            importlib.reload(eco)
        elif service == "geofabrik":
            os.environ["NOMINATIM_QUERY_URL"] = GEOFABRIK_QUERY_URL
            importlib.reload(eco)
        elif service == "OSM":
            os.environ["NOMINATIM_QUERY_URL"] = OPENSTREETMAP_QUERY_URL
            importlib.reload(eco)

Function calls one of the nominatim services with a specified URL(container):

   def test_geofabrik_and_nominatim(self):
        lat, lon = 41.8239891, -71.4128343
        NominatimTest.nominatim("container")
        container_result = eco.Geocoder.get_json_reverse(lat,lon)

Nominatim.py gets environment variable that was set in the test, and uses it in whatever function is called:

 NOMINATIM_QUERY_URL = os.environ.get("NOMINATIM_QUERY_URL")
...
query_url = NOMINATIM_QUERY_URL + "/reverse?"

shankari mentioned this issue Aug 15, 2023

Creating TestNominatim.py e-mission/e-mission-server#929

Merged

MukuFlash03 mentioned this issue Mar 5, 2024

Unify buid and deploy processes across the various components of OpenPATH #1048

Open

⚡ 🚸 Re-enable populating addresses in trips and make sure that they are used on the phone #937

⚡ 🚸 Re-enable populating addresses in trips and make sure that they are used on the phone #937

Comments

shankari commented Jul 14, 2023

shankari commented Jul 14, 2023

shankari commented Jul 15, 2023

nataliejschultz commented Jul 18, 2023

shankari commented Jul 18, 2023

shankari commented Jul 18, 2023

shankari commented Jul 18, 2023

shankari commented Jul 18, 2023

nataliejschultz commented Jul 20, 2023

nataliejschultz commented Aug 2, 2023 • edited Loading

shankari commented Aug 2, 2023 • edited Loading

shankari commented Aug 4, 2023

nataliejschultz commented Aug 7, 2023

nataliejschultz commented Aug 8, 2023

nataliejschultz commented Aug 8, 2023

shankari commented Aug 8, 2023

shankari commented Aug 8, 2023 • edited Loading

nataliejschultz commented Aug 8, 2023 • edited Loading

shankari commented Aug 8, 2023 • edited Loading

shankari commented Aug 9, 2023

nataliejschultz commented Aug 9, 2023

nataliejschultz commented Aug 9, 2023

nataliejschultz commented Aug 9, 2023

nataliejschultz commented Aug 9, 2023 • edited Loading

shankari commented Aug 9, 2023

shankari commented Aug 9, 2023 • edited Loading

shankari commented Aug 9, 2023

nataliejschultz commented Aug 11, 2023

nataliejschultz commented Aug 15, 2023

nataliejschultz commented Aug 15, 2023

nataliejschultz commented Aug 28, 2023

nataliejschultz commented Aug 30, 2023

shankari commented Aug 30, 2023

shankari commented Aug 30, 2023

shankari commented Aug 30, 2023

nataliejschultz commented Sep 11, 2023

nataliejschultz commented Sep 12, 2023

shankari commented Sep 12, 2023

shankari commented Sep 12, 2023

nataliejschultz commented Sep 14, 2023

nataliejschultz commented Sep 19, 2023

nataliejschultz commented Sep 19, 2023

nataliejschultz commented Sep 19, 2023

shankari commented Sep 19, 2023 • edited Loading

nataliejschultz commented Sep 20, 2023

shankari commented Sep 20, 2023

nataliejschultz commented Sep 21, 2023

nataliejschultz commented Sep 21, 2023

nataliejschultz commented Sep 26, 2023

shankari commented Sep 26, 2023

nataliejschultz commented Sep 26, 2023 • edited Loading

shankari commented Sep 26, 2023 • edited Loading

nataliejschultz commented Sep 26, 2023

shankari commented Sep 26, 2023

nataliejschultz commented Sep 27, 2023

nataliejschultz commented Sep 28, 2023

nataliejschultz commented Sep 29, 2023

nataliejschultz commented Oct 2, 2023

nataliejschultz commented Oct 9, 2023 • edited Loading

Set environment variables for web-server service in docker-compose:

TestNominatim.py gets these variables:

Nominatim module sets NOMINATIM_QUERY_URL to one of the URLs, based on how it's called:

Function calls one of the nominatim services with a specified URL(container):

Nominatim.py gets environment variable that was set in the test, and uses it in whatever function is called:

nataliejschultz commented Aug 2, 2023 •

edited

Loading

shankari commented Aug 2, 2023 •

edited

Loading

shankari commented Aug 8, 2023 •

edited

Loading

nataliejschultz commented Aug 8, 2023 •

edited

Loading

shankari commented Aug 8, 2023 •

edited

Loading

nataliejschultz commented Aug 9, 2023 •

edited

Loading

shankari commented Aug 9, 2023 •

edited

Loading

shankari commented Sep 19, 2023 •

edited

Loading

nataliejschultz commented Sep 26, 2023 •

edited

Loading

shankari commented Sep 26, 2023 •

edited

Loading

nataliejschultz commented Oct 9, 2023 •

edited

Loading