Skip to content
"Münster-Info-Hub" - Ein Projekt vom MÜNSTERHACK 2019
HTML Python JavaScript Other
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
elasticsearch
html
import
.gitignore
LICENSE
README.md

README.md

muenster-info-hub

Project info

Created during MünsterHack 2019. Searchengine and open API for Münster related events.

Directories:

Start developing

start elasticsearch

cd elasticsearch
export COMPOSE_PROJECT_NAME="msinfohub"
export ELASTICSEARCH_URL_PREFIX="http://elasticsearch:9200/msinfohub-"

docker-compose up
# optional

# ip for elasticsearch
export ELASTICSEARCH_IP=$(docker network inspect msinfohub_default | jq -r '.[].Containers | to_entries[] | select(.value.Name=="'"$COMPOSE_PROJECT_NAME"'_elasticsearch_1") | .value.IPv4Address | split("/")[0]')
echo "xdg-open http://$ELASTICSEARCH_IP:9200"

# ip for kibana
export KIBANA_IP=$(docker network inspect msinfohub_default | jq -r '.[].Containers | to_entries[] | select(.value.Name=="'"$COMPOSE_PROJECT_NAME"'_kibana_1") | .value.IPv4Address | split("/")[0]')
echo "xdg-open http://$KIBANA_IP:5601"
# delete index
docker run --network msinfohub_default curlimages/curl:7.65.3 \
  curl --request DELETE "$ELASTICSEARCH_URL_PREFIX*"

# put mapping template
cd ./mapping-template
docker-compose up --build

# import meinestadt
cd ./meinestadt
docker-compose up --build

# import-events-muenster.de
export MAPQUEST_KEY="FIXME"
cd ./import-events-muenster.de
docker-compose up --build

# import partyphase and virtualnights
cd ./party_scraper
docker-compose up --build ODER
SCRAPE_START=yyyy-mm-dd SCRAPE_END=yyyy-mm-dd docker-compose up --build

# import wn
cd ./import/wn
docker-compose up --build

Datenstruktur

{
    "title", 
    "subtitle", // (optional) 
    "start_date": "2008-03-01T13:00:00Z", 
    "end_date": "2008-03-01T15:00:00Z",  // (optional)
    "description", // (optional)
    "link", 
    "category", // (optional)
    "location_name", 
    "location_address",  // (optional)
    "source", // origin of these data
    "tags": [ // (optional)
        "bla", ...
    ],
    "geo": {
        "lat": "51.956944",
        "lon": "7.005556"
        },
    "images":[
        {
        "image_url",  // (optional)
        "image_text", // (optional)
        "image_copyright"    // (optional)
        }
    ],  // (optional)
}

Elasticsearch/API debuggen:

The index name that we currently use in production is: msinfohub-events

Elasticsearch Query Anleitung

Man muss nicht unbedingt komplexe JSON-Queries an Elasticsearch schicken, sondern man kann auch viel durch simple GET-Requests erreichen.

Wie die Abfragesprache im Parameter “q” funktioniert: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html Hier steht wie man nur einzelne Felder zurückbeommt: https://www.elastic.co/guide/en/elasticsearch/reference/7.3/docs-get.html#docs-get

Beispiel-Query:

https://api.muenster.jetzt/infohub/_search?_source=source,geo,start_date&size=200&q=start_date:%3Enow

Erklärung der Beispiel-Query:

  • Parameter "q" fragt in diesem Fall nur Events ab, die start_date in der Zukunft haben
  • Wegen "_source"-Parameter werden nur die Felder source, geo, start_date zurückgegeben
  • Durch "size" werden maximal 200 results returnt

Elasticsearch fortgeschrittene Aktionen

Query per curl absenden mit der suche nach einem bestimmten Wert in einem bestimmten Feld:

curl -X POST https://api.muenster.jetzt/msinfohub-events/_search?pretty -H 'Content-Type: application/json' -d'{"query":{"term":{"source":"meinestadt.de"}}}'

# Alle Einträge einer Eventquelle löschen:
curl -X POST https://api.muenster.jetzt/msinfohub-events/_delete_by_query?pretty -H 'Content-Type: application/json' -d'{"query":{"term":{"source":"meinestadt.de"}}}'
You can’t perform that action at this time.