Skip to content
Permalink
Browse files

Entrypoint fix for docker (#1041)

* fix#982

* update

* Handle Burp Collaborator issues

This is an enhancement in order to handle findings that contain Burp collaborator interactions which have additional request/response pairs and additional information that was normally ignored or breaking the importing of the report .

* Fix flake8 errors

* More accessibility fixes for top and side menu

* Made search button accessible

* tune uwsgi read timeout (troubleshoots 'upstream timed out' on POST import_scan_results)

* add a different docker-compose for dev and for release

* add a different docker-compose for dev and for release

* Update README.md

* cicd engagement: after delete return to cicd engagement view

* Updated DOCKER.md in dev branch to include info on checking versions for docker and docker-compose. People are getting tripped up by installs that aren't up to date

* Docker readme update

* Travis compose sourcing script

* Github template DB addition, docker sourcing
  • Loading branch information...
aaronweaver committed Apr 12, 2019
1 parent 0380a90 commit b36f03b2ebc38366fc4cc34fbdb76ba5a2518b7e
@@ -2,7 +2,7 @@
name: Bug report
about: Create a report to help us improve
title: ''
labels: ''
labels: 'bug'
assignees: ''

---
@@ -2,7 +2,7 @@
name: Security issue
about: Report a security issue
title: Please submit via our security reporting program, not GitHub
labels: ''
labels: 'security'
assignees: ''

---
@@ -6,8 +6,23 @@ use the [Helm and Kubernetes](KUBERNETES.md) approach.

## Setup via Docker Compose

If you start your DefectDojo instance on Docker Compose for the first time, just
run `docker-compose up`.
To start your DefectDojo instance on Docker Compose for the first time, just
run:

```zsh
. docker/aliases_release.sh
docker-compose up
```

or

```zsh
docker-compose -f docker-compose_base.yml -f docker-compose_uwsgi-release.yml up
```

This command will run the application based on images commited on dockerhub (or the last images built locally). If you need to be more up to date, see "Build images locally" below

**NOTE:** Installing with docker-compose requires the latest version of docker and docker-compose - at least docker 18.09.4 and docker-compose 1.24.0. See "Checking Docker versions" below for version errors during running docker-compose up.

**NOTE:** Installing with docker-compose requires the latest version of docker and docker-compose - at least docker 18.09.4 and docker-compose 1.24.0. See "Checking Docker versions" below for version errors during running docker-compose up.

@@ -22,17 +37,34 @@ container_id=(`docker ps -a \
docker logs $container_id 2>&1 | grep "Admin password:"
```

or:

```zsh
docker logs django-defectdojo_initializer_1
```

If you ran DefectDojo with compose before and you want to prevent the
initializer container from running again, define an environment variable
`DD_INITIALIZE=false` to prevent re-initialization.
DD_INITIALIZE=false to prevent re-initialization.

### Develop with Docker Compose

For developing the easiset way to make changes is to startup DefectDojo in debug by running `docker-compose -f docker-compose.yml up`. This starts the DefectDojo (uwsgi) container with manage.py and shares the local source directory so that changes to the code immediately restart the process.
For developing the easiset way to make changes is to startup DefectDojo in debug by running:

Navigate to the container directly, <http://localhost:8000>
```zsh
. docker/aliases_dev.sh
docker-compose up
```

or

```zsh
docker-compose -f docker-compose_base.yml -f docker-compose_uwsgi-dev.yml up
```

This starts the DefectDojo (uwsgi) container with manage.py and shares the local source directory so that changes to the code immediately restart the process.

Build the containers locally: `docker-compose build`
Navigate to the container directly, <http://localhost:8000>

The initializer container can be disabled by exporting: `export DD_INITIALIZE=false`

@@ -41,14 +73,19 @@ The initializer container can be disabled by exporting: `export DD_INITIALIZE=fa
Build the docker containers locally for testing purposes.

```zsh
# Build Dev Compose
docker-compose build
or:
# Build images
docker build -t defectdojo/defectdojo-django -f Dockerfile.django .
docker build -t defectdojo/defectdojo-nginx -f Dockerfile.nginx .
```

### Clean up Docker Compose

Removes all containers
Removes all containers

```zsh
docker-compose down
@@ -4,4 +4,5 @@ When submitting a pull request, please make sure you have completed the followin

- [ ] Your code is flake8 compliant (DefectDojo's code isn't currently flake8 compliant, but we're trying to correct that.)
- [ ] If this is a new feature and not a bug fix, you've included the proper documentation in the ReadTheDocs documentation folder. https://github.com/DefectDojo/Documentation/tree/master/docs or provide feature documentation in the PR.
- [ ] Model changes should include the necessary migrations in the dojo/dd_migrations folder.
- [ ] Add applicable tests to the unit tests.
@@ -98,6 +98,7 @@ DefectDojo is maintained by:
Project Moderators can help you with pull requests or feedback on dev ideas.

* [Alex Dracea](https://www.linkedin.com/in/alexandru-marin-dracea-910b51122/)
* [Valentijn Scholten](https://www.linkedin.com/in/valentijn-scholten/)

## Hall of Fame

@@ -10,24 +10,6 @@ services:
published: ${DD_PORT:-8080}
protocol: tcp
mode: host
uwsgi:
image: defectdojo/defectdojo-django:latest
build:
context: ./
dockerfile: Dockerfile.django
depends_on:
- mysql
entrypoint: ['/wait-for-it.sh', 'mysql:3306', '-t', '30', '--', '/entrypoint-uwsgi-dev.sh']
volumes:
- '.:/app'
ports:
- 8000:8000
environment:
DD_DEBUG: 'on'
DD_ALLOWED_HOSTS: ${DD_ALLOWED_HOSTS:-*}
DD_DATABASE_URL: ${DD_DATABASE_URL:-mysql://defectdojo:defectdojo@mysql:3306/defectdojo}
DD_CELERY_BROKER_USER: ${DD_CELERY_BROKER_USER:-guest}
DD_CELERY_BROKER_PASSWORD: ${DD_CELERY_BROKER_USER:-guest}
celerybeat:
image: defectdojo/defectdojo-django:latest
depends_on:
@@ -62,8 +44,6 @@ services:
DD_INITIALIZE: ${DD_INITIALIZE:-true}
mysql:
image: mysql:5.7
volumes:
- defectdojo_data:/var/lib/mysql
environment:
MYSQL_RANDOM_ROOT_PASSWORD: 'yes'
DD_DATABASE_URL: ${DD_DATABASE_URL:-mysql://defectdojo:defectdojo@mysql:3306/defectdojo}
@@ -73,5 +53,3 @@ services:
command: ['mysqld', '--character-set-server=utf8mb4', '--collation-server=utf8mb4_unicode_ci']
rabbitmq:
image: rabbitmq:3.7
volumes:
defectdojo_data: {}
@@ -0,0 +1,20 @@
---
version: '3.7'
services:
uwsgi:
build:
context: ./
dockerfile: Dockerfile.django
depends_on:
- mysql
entrypoint: ['/wait-for-it.sh', 'mysql:3306', '-t', '30', '--', '/entrypoint-uwsgi-dev.sh']
volumes:
- '.:/app'
ports:
- 8000:8000
environment:
DD_DEBUG: 'on'
DD_ALLOWED_HOSTS: ${DD_ALLOWED_HOSTS:-*}
DD_DATABASE_URL: ${DD_DATABASE_URL:-mysql://defectdojo:defectdojo@mysql:3306/defectdojo}
DD_CELERY_BROKER_USER: ${DD_CELERY_BROKER_USER:-guest}
DD_CELERY_BROKER_PASSWORD: ${DD_CELERY_BROKER_USER:-guest}
@@ -7,8 +7,8 @@ services:
- mysql
entrypoint: ['/wait-for-it.sh', 'mysql:3306', '-t', '30', '--', '/entrypoint-uwsgi.sh']
environment:
DD_DEBUG: 'on'
DD_DEBUG: 'off'
DD_ALLOWED_HOSTS: ${DD_ALLOWED_HOSTS:-*}
DD_DATABASE_URL: ${DD_DATABASE_URL:-mysql://defectdojo:defectdojo@mysql:3306/defectdojo}
DD_CELERY_BROKER_USER: ${DD_CELERY_BROKER_USER:-guest}
DD_CELERY_BROKER_PASSWORD: ${DD_CELERY_BROKER_USER:-guest}
DD_CELERY_BROKER_PASSWORD: ${DD_CELERY_BROKER_USER:-guest}
@@ -0,0 +1 @@
alias docker-compose='docker-compose -f docker-compose_base.yml -f docker-compose_uwsgi-dev.yml'
@@ -0,0 +1 @@
alias docker-compose='docker-compose -f docker-compose_base.yml -f docker-compose_uwsgi-release.yml'
@@ -6,3 +6,4 @@ exec uwsgi \
"--${DD_UWSGI_MODE}" "${DD_UWSGI_ENDPOINT}" \
--protocol uwsgi \
--wsgi dojo.wsgi:application
--http-timeout 1800
@@ -223,7 +223,11 @@ def delete_engagement(request, eid):
messages.SUCCESS,
'Engagement and relationships removed.',
extra_tags='alert-success')
return HttpResponseRedirect(reverse("view_engagements", args=(product.id, )))

if engagement.engagement_type == 'CI/CD':
return HttpResponseRedirect(reverse("view_engagements_cicd", args=(product.id, )))
else:
return HttpResponseRedirect(reverse("view_engagements", args=(product.id, )))

collector = NestedObjects(using=DEFAULT_DB_ALIAS)
collector.collect([engagement])
@@ -1311,6 +1311,7 @@ def merge_finding_product(request, pid):
finding_to_merge_into = form.cleaned_data['finding_to_merge_into']
findings_to_merge = form.cleaned_data['findings_to_merge']
finding_descriptions = ''
finding_references = ''
notes_entry = ''
static = False
dynamic = False
@@ -1337,6 +1338,10 @@ def merge_finding_product(request, pid):
if finding.file_path:
finding_descriptions = "{}\n**File Path:** {}\n".format(finding_descriptions, finding.file_path)

# If checked merge the Reference
if form.cleaned_data['append_reference']:
finding_references = "{}\n{}".format(finding_references, finding.references)

# if checked merge the endpoints
if form.cleaned_data['add_endpoints']:
finding_to_merge_into.endpoints.add(*finding.endpoints.all())
@@ -1377,6 +1382,9 @@ def merge_finding_product(request, pid):
if finding_to_merge_into.file_path is None:
file_path = finding_to_merge_into.file_path

if finding_references != '':
finding_to_merge_into.references = "{}\n{}".format(finding_to_merge_into.references, finding_references)

finding_to_merge_into.static_finding = static
finding_to_merge_into.dynamic_finding = dynamic

@@ -386,6 +386,9 @@ class MergeFindings(forms.ModelForm):
mark_tag_finding = forms.BooleanField(label="Tag Merged Finding", initial=True, required=False,
help_text="Creates a tag titled 'merged' for the finding that will be merged. If the 'Finding Action' is set to 'inactive' the inactive findings will be tagged with 'merged-inactive'.")

append_reference = forms.BooleanField(label="Append Reference", initial=True, required=False,
help_text="Reference in all findings will be appended into the merged finding.")

finding_action = forms.ChoiceField(
required=True,
choices=FINDING_ACTION,
@@ -405,11 +408,11 @@ def __init__(self, *args, **kwargs):
queryset=findings, required=True, label="Findings to Merge",
widget=forms.widgets.SelectMultiple(attrs={'size': 10}),
help_text=('Select the findings to merge.'))
self.fields.keyOrder = ['finding_to_merge_into', 'findings_to_merge', 'append_description', 'add_endpoints']
self.fields.keyOrder = ['finding_to_merge_into', 'findings_to_merge', 'append_description', 'add_endpoints', 'append_reference']

class Meta:
model = Finding
fields = ['append_description', 'add_endpoints']
fields = ['append_description', 'add_endpoints', 'append_reference']


class UploadRiskForm(forms.ModelForm):
@@ -75,29 +75,29 @@
<div class="input-group">
<input id="simple_search" type="text" name="query" class="form-control"
placeholder="Search...">
<span class="input-group-btn">
<button class="btn btn-primary" type="submit">
<i class="fa fa-search"></i>
</button>
</span>
<span class="input-group-btn">
<button class="btn btn-primary" type="submit" aria-label="Search">
<i class="fa fa-search"></i>
</button>
</span>
</div>
</form>
</div>
<!-- /input-group -->
</li>
<li class="dropdown">
<a class="dropdown-toggle dropdown-toggle-h{% alert_count %}" data-toggle="dropdown" href="#" aria-label="{% alert_count %} Alerts">
<i class="fa fa-bell fa-fw"></i><span
id="alert_count" class="badge badge-count badge-count{% alert_count %}">{% alert_count %}</span>
<i class="fa fa-caret-down"></i>
</a>
<ul class="dropdown-menu dropdown-alerts">
</ul>
<!-- /.dropdown-alerts -->
</li>
<li class="dropdown">
<a class="dropdown-toggle dropdown-toggle-h{% alert_count %}" data-toggle="dropdown" href="#" aria-expanded="false" aria-label="{% alert_count %} Alerts">
<i class="fa fa-bell fa-fw"></i>
<span id="alert_count" class="badge badge-count badge-count{% alert_count %}">{% alert_count %}</span>
<i class="fa fa-caret-down"></i>
</a>
<ul class="dropdown-menu dropdown-alerts">
</ul>
<!-- /.dropdown-alerts -->
</li>
{% endif %}
<li class="dropdown">
<a class="dropdown-toggle" data-toggle="dropdown" href="#" aria-label="User Menu">
<a class="dropdown-toggle" data-toggle="dropdown" href="#" aria-expanded="false" aria-label="User Menu">
<i class="fa fa-user fa-fw"></i> <i class="fa fa-caret-down"></i>
</a>
<ul class="dropdown-menu dropdown-user">
@@ -131,7 +131,10 @@

{% if request.user.is_staff %}
<li>
<a href="{% url 'dashboard' %}" aria-label="Dashboard"><i class="fa fa-dashboard fa-fw"></i><span>Dashboard</span></a>
<a href="{% url 'dashboard' %}" aria-disabled="true" aria-expanded="false" aria-label="Dashboard">
<i class="fa fa-dashboard fa-fw"></i>
<span>Dashboard</span>
</a>
</li>
{% endif %}
<li>
@@ -192,6 +192,35 @@ def get_item(item_node, test):
except:
response = ""
unsaved_req_resp.append({"req": request, "resp": response})
collab_details = list()
collab_text = None
for event in item_node.findall('./collaboratorEvent'):
collab_details.append(event.findall('interactionType')[0].text)
collab_details.append(event.findall('originIp')[0].text)
collab_details.append(event.findall('time')[0].text)
if collab_details[0] == 'DNS':
collab_details.append(event.findall('lookupType')[0].text)
collab_details.append(event.findall('lookupHost')[0].text)
collab_text = "The Collaborator server received a " + collab_details[0] + " lookup of type " + collab_details[3] + \
" for the domain name " + \
collab_details[4] + " at " + collab_details[2] + \
" originating from " + collab_details[1] + " ."

for request_response in event.findall('./requestresponse'):
try:
request = request_response.findall('request')[0].text
except:
request = ""
try:
response = request_response.findall('response')[0].text
except:
response = ""
unsaved_req_resp.append({"req": request, "resp": response})
if collab_details[0] == 'HTTP':
collab_text = "The Collaborator server received an " + \
collab_details[0] + " request at " + collab_details[2] + \
" originating from " + collab_details[1] + " ."


try:
dupe_endpoint = Endpoint.objects.get(
@@ -248,6 +277,8 @@ def get_item(item_node, test):
detail = do_clean(item_node.findall('issueDetail'))
if detail:
detail = text_maker.handle(detail)
if collab_text:
detail = text_maker.handle(detail + '<p>' + collab_text + '</p>')

remediation = do_clean(item_node.findall('remediationBackground'))
if remediation:
@@ -28,6 +28,7 @@ http {
location / {
include /run/uwsgi_pass;
include /etc/nginx/wsgi_params;
uwsgi_read_timeout 1800;
}
error_page 500 502 503 504 /50x.html;
}
@@ -134,14 +134,14 @@ echo "Running test ${TEST}"
docker)
echo "Validating docker compose"
build_containers
docker-compose up -d
docker-compose -f docker-compose_base.yml -f docker-compose_uwsgi-release.yml up -d
echo "Waiting for services to start"
# Wait for services to become available
sleep 80
echo "Testing DefectDojo Service"
curl -s -o "/dev/null" http://localhost:8080 -m 120
echo "Docker compose container status"
docker-compose ps
docker-compose -f docker-compose_base.yml -f docker-compose_uwsgi-release.yml ps
;;
snyk)
echo "Snyk security testing on containers"

0 comments on commit b36f03b

Please sign in to comment.
You can’t perform that action at this time.