New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document internals #213

Merged
merged 9 commits into from Sep 13, 2015

Conversation

Projects
None yet
2 participants
@migurski
Member

migurski commented Sep 11, 2015

@NelsonMinar, this is ready for your review.

Closes #203.

@NelsonMinar

This comment has been minimized.

Show comment
Hide comment
@NelsonMinar

NelsonMinar Sep 11, 2015

Contributor

Hey this is awesome! Nice job writing this up. I think other folks can use this to understand the system pretty well, and will also serve to remind us how this all works. I particularly love all the links to stuff. The docs will likely fall out of date without diligence, but that's kind of unavoidable.

Some suggested additions:

In the components doc, it'd be nice to have a sentence explaining the product purpose of each component.
web hook: "It is responsible for running jobs on demand in response to pull requests and changes in the GitHub repository.
worker: "It does the actual work of running a source and producing output files."

dequeue: "It collects the results from jobs that are completed and reports back to GitHub"


Some other comments:
worker: Also clarify that the worker process is single threaded, but is persistent and keeps looking for work to do.
scheduled tasks: where does the cron job live, on the web hook machine?

In the persistence doc:

Database: mention backups?

Queue: mention that in steady state, the queue is empty? Ie no permanent data, only temporary data for running tasks.

In the processes doc:

Would be nice to document what starts / monitors these processes.

Should include the flask server as a process?

Contributor

NelsonMinar commented Sep 11, 2015

Hey this is awesome! Nice job writing this up. I think other folks can use this to understand the system pretty well, and will also serve to remind us how this all works. I particularly love all the links to stuff. The docs will likely fall out of date without diligence, but that's kind of unavoidable.

Some suggested additions:

In the components doc, it'd be nice to have a sentence explaining the product purpose of each component.
web hook: "It is responsible for running jobs on demand in response to pull requests and changes in the GitHub repository.
worker: "It does the actual work of running a source and producing output files."

dequeue: "It collects the results from jobs that are completed and reports back to GitHub"


Some other comments:
worker: Also clarify that the worker process is single threaded, but is persistent and keeps looking for work to do.
scheduled tasks: where does the cron job live, on the web hook machine?

In the persistence doc:

Database: mention backups?

Queue: mention that in steady state, the queue is empty? Ie no permanent data, only temporary data for running tasks.

In the processes doc:

Would be nice to document what starts / monitors these processes.

Should include the flask server as a process?

@migurski

This comment has been minimized.

Show comment
Hide comment
@migurski

migurski Sep 12, 2015

Member

I’ve incorporated your feedback. I didn’t include Flask as a process, since I was thinking of that doc more as “narratives” rather than processes in the Unix sense. Does that make sense?

Member

migurski commented Sep 12, 2015

I’ve incorporated your feedback. I didn’t include Flask as a process, since I was thinking of that doc more as “narratives” rather than processes in the Unix sense. Does that make sense?

@NelsonMinar

This comment has been minimized.

Show comment
Hide comment
@NelsonMinar

NelsonMinar Sep 13, 2015

Contributor

I see what you mean about processes, and the CI process is documented which covers most of what the Flask server is doing anyway.

Thanks again for writing all this up!

Contributor

NelsonMinar commented Sep 13, 2015

I see what you mean about processes, and the CI process is documented which covers most of what the Flask server is doing anyway.

Thanks again for writing all this up!

migurski added a commit that referenced this pull request Sep 13, 2015

@migurski migurski merged commit aab57f9 into master Sep 13, 2015

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@migurski migurski deleted the document-internals-#203 branch Sep 13, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment