Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add job type to elasticsearch logs #948

Closed
mikenholt opened this issue Sep 5, 2017 · 1 comment · Fixed by #1047
Closed

Add job type to elasticsearch logs #948

mikenholt opened this issue Sep 5, 2017 · 1 comment · Fixed by #1047
Assignees
Labels

Comments

@mikenholt
Copy link
Contributor

When debugging it would be ideal to have the job_type id associated with logs in elasticsearch to help pinpoint potential errors.

@dfaller dfaller added the logging label Sep 7, 2017
@dfaller dfaller added this to the Backlog milestone Sep 7, 2017
@gisjedi
Copy link
Contributor

gisjedi commented Dec 13, 2017

Job and JobType models can be found in scale/job/models.py

  1. The JobType.id needs to be added as a tag where Docker parameters are applied in job/configuration/configurators.py under _configure_all_tasks.py roughly around line 285
  2. Build of scale image and push to personal Docker Hub account.
  3. Deploy into cluster and once all green, change scale-logstash service in Marathon to set LOGSTASH_DEBUG to true. This will redeploy the logstash service and the messages received can be seen in the STDOUT logs.
  4. Once verified tags applied from step 1 are arriving, identify updates needed to the logstash config to extract JobType IDs.
  5. The logging forwarder is responsible for pulling the data applied to the tags on the Docker container. This can be found in the dockerfiles/logstash/logstash.conf-template file.
  6. The Logstash config can be quickly iterated on by using the TEMPLATE_URI environment variable to scale-logstash service. Hosting on https://gist.github.com is likely the quickest way to make that public accessible.

@Fizz11 Fizz11 self-assigned this Dec 18, 2017
Fizz11 added a commit that referenced this issue Dec 20, 2017
#948 Add job type to elasticsearch logs
@dfaller dfaller modified the milestones: Backlog, Sprint 12-18-2017 Dec 20, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants