Skip to content

Commit

Permalink
Bug 1630293 - Docker, docs and logging changes (#6665)
Browse files Browse the repository at this point in the history
* Docker: Bug 1630293 - Increase max MySql connections

* docs: Set a concurrency of 1

* docs: Remove the usage of -B for Docker set up

Using -B in more than one instance will cause the remaining celery beats to
execute in more than one container.

* Reduce unsupported known failure line while still reporting to New Relic
  • Loading branch information
armenzg committed Jul 30, 2020
1 parent cb16205 commit c9b86aa
Show file tree
Hide file tree
Showing 4 changed files with 7 additions and 8 deletions.
2 changes: 1 addition & 1 deletion docker/mysql.cnf
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
[mysqld]
character_set_server="utf8"
collation_server="utf8_bin"
max_connections=300
max_connections=1000

# Ensure operations involving astral characters fail loudly,
# rather than mysql silently replacing each each byte of the
Expand Down
5 changes: 1 addition & 4 deletions docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,12 +148,9 @@ Ingestion tasks populate the database with version control push logs, queued/run
- Start up a celery worker to process async tasks:

```bash
docker-compose run backend celery -A treeherder worker -B --concurrency 5
docker-compose run backend celery -A treeherder worker --concurrency 1
```

The "-B" option tells the celery worker to startup a beat service, so that periodic tasks can be executed.
You only need one worker with the beat service enabled. Multiple beat services will result in periodic tasks being executed multiple times.

- Then in a new terminal window, run `docker-compose run backend bash`, and follow the steps from the [loading pulse data](pulseload.md) page.

### Manual ingestion
Expand Down
4 changes: 2 additions & 2 deletions docs/pulseload.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,15 +51,15 @@ you would like to process.
Open a new terminal window. To run all the queues do:

```bash
docker-compose run backend celery -A treeherder worker -B --concurrency 5
docker-compose run backend celery -A treeherder worker --concurrency 1
```

You will see a list of activated queues. If you wanted to narrow that down, then note
which queues you'd like to run and add them to a comma-separated list. For instance, to
only do Log Parsing:

```bash
celery -A treeherder worker -B -Q log_parser,log_parser_fail --concurrency 5
celery -A treeherder worker -Q log_parser,log_parser_fail --concurrency 1
```

## Posting Data
Expand Down
4 changes: 3 additions & 1 deletion treeherder/log_parser/failureline.py
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,9 @@ def create(job_log, log_list):
action = failure_line['action']
if action not in FailureLine.ACTION_LIST:
newrelic.agent.record_custom_event("unsupported_failure_line_action", failure_line)
logger.exception(ValueError(f'Unsupported FailureLine ACTION: {action}'))
# Unfortunately, these two errors flod the logs
if action not in ['group_result', 'test_groups']:
logger.exception(ValueError(f'Unsupported FailureLine ACTION: {action}'))
failure_lines = [
create_failure_line(job_log, failure_line)
for failure_line in log_list
Expand Down

0 comments on commit c9b86aa

Please sign in to comment.