Docker container doesn't sync data-configuration.yml on start of OpenTAXII image

[vnktsh]

Hello guys,

I pulled latest docker image from eclecticiq/opentaxii:latest (0107b7d07fd2). Started the container with default configuration file as present in git repo under examples folder. But the problem is, taxii-server isn't being configured with the settings present data-configuration.yml. One needs to manually sync the data inside the container. According to Dockerfile and entrypoint.sh script, it should be configured while booting up.

I ran container with following commands

❯ pwd
/home/vnktsh/projects/OpenTAXII/examples

❯ ls
create-fixtures.sh  data-configuration.yml  docker-compose.yml  hooks.py  Makefile  stix

❯ docker run --rm -v $PWD:/input:ro -p 9000:9000 --name "taxii" eclecticiq/opentaxii:latest
Using config: 
---

domain: "localhost:9000"

persistence_api:
  class: opentaxii.persistence.sqldb.SQLDatabaseAPI
  parameters:
    db_connection: sqlite:////data/data.db
    create_tables: yes

auth_api:
  class: opentaxii.auth.sqldb.SQLDatabaseAPI
  parameters:
    db_connection: sqlite:////data/auth.db
    create_tables: yes
    secret: notVerySecret

logging:
  opentaxii: info
  root: info

hooks:
{"api": "opentaxii.persistence.sqldb.SQLDatabaseAPI", "event": "api.initialized", "logger": "opentaxii.utils", "level": "info", "timestamp": "2019-01-29T02:59:22.807415Z"}
{"api": "opentaxii.auth.sqldb.SQLDatabaseAPI", "event": "api.initialized", "logger": "opentaxii.utils", "level": "info", "timestamp": "2019-01-29T02:59:22.863277Z"}
{"event": "opentaxii.server_configured", "logger": "opentaxii.server", "level": "info", "timestamp": "2019-01-29T02:59:22.864190Z"}
{"event": "Starting gunicorn 19.9.0", "logger": "gunicorn.error", "level": "info", "timestamp": "2019-01-29T02:59:22.916027Z"}
{"event": "Listening at: http://0.0.0.0:9000 (1)", "logger": "gunicorn.error", "level": "info", "timestamp": "2019-01-29T02:59:22.917902Z"}
{"event": "Using worker: sync", "logger": "gunicorn.error", "level": "info", "timestamp": "2019-01-29T02:59:22.918415Z"}
{"event": "Booting worker with pid: 16", "logger": "gunicorn.error", "level": "info", "timestamp": "2019-01-29T02:59:22.924967Z"}
{"event": "Booting worker with pid: 17", "logger": "gunicorn.error", "level": "info", "timestamp": "2019-01-29T02:59:23.037134Z"}

Container starts without any errors.
However when I try to fetch discovery service using Cabby (taxii client) it throws 404 error

❯ taxii-discovery --host localhost --port 9000 --discovery services/discovery-a
2019-01-28 21:05:10,709 INFO: Sending Discovery_Request to http://localhost:9000/services/discovery-a
2019-01-28 21:05:10,785 ERROR: HTTP Error: status code 404

If we go into the running container I find there exists config file in input directory, and the script entrypoint.sh successfully executed opentaxii-sync-data command but the database has no entries in its tables.

❯ docker exec -it taxii "/bin/bash"
root@93fc3fccbe96:/# ls
bin   data  entrypoint.sh  home   lib	 media	opentaxii.yml  proc  run   srv	tmp  var
boot  dev   etc		   input  lib64  mnt	opt	       root  sbin  sys	usr  venv 
root@93fc3fccbe96:/# ls /input/
create-fixtures.sh  data-configuration.yml  docker-compose.yml  hooks.py  Makefile  stix
root@93fc3fccbe96:/# ls /tmp/
auth.db  data.db  opentaxii.yml
root@93fc3fccbe96:/# sqlite3 /tmp/data.db 
SQLite version 3.16.2 2017-01-06 16:32:41
Enter ".help" for usage hints.
sqlite> .tables
collection_to_content_block  result_sets                
content_blocks               service_to_collection      
data_collections             services                   
inbox_messages               subscriptions              
sqlite> select * from services;
sqlite> select * from data_collections;

If you see above "select * from services;" return empty table.

What resolved the issue is manually syncing the data-configuration.yml inside container.

root@93fc3fccbe96:/# [ -f /input/data-configuration.yml ] && OPENTAXII_CONFIG=$OPENTAXII_CONFIG opentaxii-sync-data -f /input/data-configuration.yml
2019-01-29T03:16:12.131047Z [opentaxii.utils] info: api.initialized {api=opentaxii.persistence.sqldb.SQLDatabaseAPI}
2019-01-29T03:16:12.174448Z [opentaxii.utils] info: api.initialized {api=opentaxii.auth.sqldb.SQLDatabaseAPI}
2019-01-29T03:16:12.175265Z [opentaxii.server] info: opentaxii.server_configured 
2019-01-29T03:16:12.210779Z [anyconfig] info: Loading: /input/data-configuration.yml 
2019-01-29T03:16:12.262724Z [opentaxii.utils] info: sync_services.updated {id=inbox_a}
2019-01-29T03:16:12.270181Z [opentaxii.utils] info: sync_services.updated {id=inbox_b}
2019-01-29T03:16:12.275061Z [opentaxii.utils] info: sync_services.updated {id=discovery_a}
2019-01-29T03:16:12.278720Z [opentaxii.utils] info: sync_services.updated {id=collection_management_a}
2019-01-29T03:16:12.283607Z [opentaxii.utils] info: sync_services.updated {id=poll_a}
2019-01-29T03:16:12.284015Z [opentaxii.utils] info: sync_services.stats {updated=5, created=0, deleted=0}
2019-01-29T03:16:12.303758Z [opentaxii.utils] info: sync_collections.updated {name=collection-a, id=1}
2019-01-29T03:16:12.314930Z [opentaxii.utils] info: sync_collections.updated {name=collection-b, id=2}
2019-01-29T03:16:12.324109Z [opentaxii.utils] info: sync_collections.updated {name=collection-c, id=3}
2019-01-29T03:16:12.340491Z [opentaxii.utils] info: sync_collections.updated {name=col-not-available, id=4}
2019-01-29T03:16:12.340828Z [opentaxii.utils] info: sync_collections.stats {updated=4, created=0, disabled=0, deleted=0}
2019-01-29T03:16:12.349108Z [opentaxii.auth.manager] warning: update_account.unknown_collection {collection=collection-xyz}
2019-01-29T03:16:12.421779Z [opentaxii.utils] info: sync_accounts.updated {username=test}
2019-01-29T03:16:12.476292Z [opentaxii.utils] info: sync_accounts.updated {username=admin}
2019-01-29T03:16:12.476604Z [opentaxii.utils] info: sync_accounts.stats {updated=2, created=0, deleted=0}

Now the entries are successfully created in sqlite DB.

root@93fc3fccbe96:/# sqlite3 /tmp/data.db 
SQLite version 3.16.2 2017-01-06 16:32:41
Enter ".help" for usage hints.
sqlite> select * from services;
2019-01-29 03:16:00.393596|inbox_a|inbox|{"address": "/services/inbox-a", "description": "Custom Inbox Service Description A", "destination_collection_required": true, "accept_all_content": true, "authentication_required": false, "protocol_bindings": ["urn:taxii.mitre.org:protocol:http:1.0"]}|2019-01-29 03:16:00.393602
2019-01-29 03:16:00.403303|inbox_b|inbox|{"address": "/services/inbox-b", "description": "Custom Inbox Service Description B", "destination_collection_required": true, "accept_all_content": false, "authentication_required": true, "supported_content": ["urn:stix.mitre.org:xml:1.1.1", "urn:custom.example.com:json:0.0.1"], "protocol_bindings": ["urn:taxii.mitre.org:protocol:http:1.0"]}|2019-01-29 03:16:00.403308
2019-01-29 03:16:00.410919|discovery_a|discovery|{"address": "/services/discovery-a", "description": "Custom Discovery Service description", "advertised_services": ["inbox_a", "inbox_b", "discovery_a", "collection_management_a", "poll_a"], "protocol_bindings": ["urn:taxii.mitre.org:protocol:http:1.0", "urn:taxii.mitre.org:protocol:https:1.0"]}|2019-01-29 03:16:00.410923
2019-01-29 03:16:00.418572|collection_management_a|collection_management|{"address": "/services/collection-management-a", "description": "Custom Collection Management Service description", "protocol_bindings": ["urn:taxii.mitre.org:protocol:http:1.0", "urn:taxii.mitre.org:protocol:https:1.0"]}|2019-01-29 03:16:00.418580
2019-01-29 03:16:00.425883|poll_a|poll|{"address": "/services/poll-a", "description": "Custom Poll Service description", "subscription_required": false, "max_result_count": 100, "max_result_size": 10, "authentication_required": true, "protocol_bindings": ["urn:taxii.mitre.org:protocol:http:1.0"]}|2019-01-29 03:16:00.425888
sqlite> 
sqlite> 

And I can fetch from taxii-client too after this.

But expected behaviour should be, the docker container while booting should sync data-configuration.yml file if it finds in \input folder. Instead it requires a manual sync inside the container.

Steps to reproduce:

• Pull latest docker image from eclecticiq/opentaxii:latest
• cd to 'examples' folder under local git repo. Mount the folder into input directory while starting the opentaxii container.
• Go inside the container, check sqlite database tables. They return empty results.

[ghost]

Hey! This might have been an oversight when we were overhauling the Dockerfile and stuff. It should be fixed now, thanks for the report!

[vnktsh]

Thanks @andreasEIQ . It works now. Any plans to update Dockerhub image?

[ghost]

It should've updated automatically, I guess some of our integrations aren't working properly.

[vnktsh]

It's updated now. Thanks a ton. Also I'm curious to know how you guys setup those integrations? Would love to explore that.

[ghost]

Docker hub provides a URL you can POST to to trigger a new build for your project, you can simply add that as a webhook in Github's repository settings.