[AIRFLOW-558] Add Support for dag.backfill=(True|False) Option by btallman · Pull Request #1830 · apache/airflow

btallman · 2016-10-08T05:43:32Z

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:

* https://issues.apache.org/jira/browse/AIRFLOW-558 *

Add Support for dag.backfill=(True|False) Option

Added a dag.backfill option and modified the scheduler to look at
the value when scheduling DagRuns (by moving dag.start_date up to
dag.previous_schedule), and added a config option scheduler.backfill_by_default (defaults to True)
that allows users to set this to False for all dags modifying the existing DAGs

In addition, we added a test to jobs.py (test_dag_backfill_option)

bolkedebruin · 2016-10-08T21:08:50Z

Overall this looks good - from a first review. However, I am not sure about having a global option in. Changing this option would fundamentally change the behavior of what airflow does and will create questions on the mailling list "why doesnt my dag backfill?".

Secondly, I would like to have this properly documented.

btallman · 2016-10-08T21:44:46Z

Good points... I will look at the documentation to figure out where to put that. As to the global option, I think as long as it is well documented and doesn't change the default it should be ok...
Ben

Get Outlook for iOS

btallman · 2016-10-09T01:04:44Z

@bolkedebruin and @r39132 and @plypaul and others, I added docs. VERY open to suggestions for improvements.

codecov-io · 2016-10-09T03:10:37Z

Current coverage is 67.01% (diff: 67.39%)

Merging #1830 into master will decrease coverage by <.01%

@@             master      #1830   diff @@
==========================================
  Files           135        135          
  Lines         10255      10297    +42   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits           6873       6901    +28   
- Misses         3382       3396    +14   
  Partials          0          0

Powered by Codecov. Last update 9df4789...f296939

r39132 · 2016-10-09T15:57:55Z

I do have some questions about this functionality as well. Is the idea that some companies don't want backfill at all? If so, I'd like to see some votes for this feature (via the JIRA).

Alternate functionality could be to have a UI feature like "pause" that is exposed on the UI per DAG and have it disabled by default (e.g. dags_are_paused_at_creation = True)? That way, it becomes easier to override.

I feel that functionality is the right way to handle this.

btallman · 2016-10-09T18:15:50Z

That actually doesn't stop backfill. As soon as you un pause the dag, it backfills.
This doesn't stop the backfill by default, only when set either in config, or on a per dag basis. Bolke and I came up with the design, and Paul said that Airbnb needs it as well. We also have several people asking on gitter for this.
In addition, this allows your own request for cron like behavior to work. This the allows us at apigee to move to the scheduler, from externally triggered dags (we use cron for all of our triggering now).
Ben

Get Outlook for iOS

bolkedebruin · 2016-10-09T20:04:00Z

Please dont mention this here. Documentation will be release together with Airflow itself, so there is no need.

r39132 · 2016-10-10T18:42:32Z

Hi!
I just tested this fix locally with a combination of DAGs. There seems to be an issue with using @hourly (and possibly the other shortcuts as well, e.g. @daily) and backfill=False. For that combination, the DAG is never scheduled!

To reproduce a case where this does not work, try

import datetime as dt

from airflow.models import DAG
from airflow.operators.dummy_operator import DummyOperator

dag = DAG(
    dag_id='simple_dummy_op_v1',
    schedule_interval='@hourly',
    backfill=False,
    start_date=dt.datetime(2016, 9, 1),
)


task1 = DummyOperator(task_id='task1', dag=dag)

However, a similar DAG with backfill=True will work :

import datetime as dt

from airflow.models import DAG
from airflow.operators.dummy_operator import DummyOperator

dag = DAG(
    dag_id='simple_dummy_w_backfill_op_v1',
    schedule_interval='@hourly',
    backfill=True,
    start_date=dt.datetime(2016, 9, 1),
)

task1 = DummyOperator(task_id='task1', dag=dag)

This suggests a problem with the combination of @hourly and backfill=False.

r39132 · 2016-10-10T23:32:09Z

It looks like the issue above has been fixed but I am now seeing an issue with a DAG that has 'depends_on_past': True, It's stuck in "RUNNING" state and does not appear to get scheduled:

The code for the problematic DAG is below :

from builtins import range
from airflow.operators.bash_operator import BashOperator
from airflow.operators.dummy_operator import DummyOperator
from airflow.models import DAG
from datetime import datetime, timedelta

seven_days_ago = datetime.combine(datetime.today() - timedelta(7),
                                  datetime.min.time())
args = {
    'owner': 'airflow',
    'depends_on_past': True,
    'start_date': seven_days_ago,
}

dag = DAG(
    dag_id='example_bash_operator_v5', default_args=args,
    schedule_interval='@hourly',
    backfill=False, 
    dagrun_timeout=timedelta(minutes=60))

cmd = 'ls -l'
run_this_last = DummyOperator(task_id='run_this_last', dag=dag)

run_this = BashOperator(
    task_id='run_after_loop', bash_command='echo 1', dag=dag)
run_this.set_downstream(run_this_last)

for i in range(3):
    i = str(i)
    task = BashOperator(
        task_id='runme_'+i,
        bash_command='echo "{{ task_instance_key_str }}" && sleep 1',
        dag=dag)
    task.set_downstream(run_this)

task = BashOperator(
    task_id='also_run_this',
    bash_command='echo "run_id={{ run_id }} | dag_run={{ dag_run }}"',
    dag=dag)
task.set_downstream(run_this_last)

if __name__ == "__main__":
    dag.cli()

btallman · 2016-10-11T03:37:29Z

Testing this change which fixes that Depends on past here:

`
@Property
@provide_session
def previous_ti(self, session=None):
""" The task instance for the task that ran before this task instance """
dag = self.task.dag

    if dag.backfill:
        return session.query(TaskInstance).filter(
            TaskInstance.dag_id == self.dag_id,
            TaskInstance.task_id == self.task.task_id,
            TaskInstance.execution_date ==
            self.task.dag.previous_schedule(self.execution_date),
        ).first()

    else:
        last_dagrun = session.query(DagRun).filter(
            DagRun.dag_id == dag.dag_id,
            DagRun.execution_date < self.execution_date
        ).order_by(
            DagRun.execution_date.desc()
        ).first()

        if last_dagrun:
            return session.query(TaskInstance).filter(
                TaskInstance.dag_id == self.dag_id,
                TaskInstance.task_id == self.task.task_id,
                TaskInstance.execution_date ==
                last_dagrun.execution_date,
            ).first()
        else:
            return None

`

r39132 · 2016-10-13T00:24:08Z

@btallman Did you test your fix against my attached code? Because, it still doesn't work against my provided example. I am resetting the DB before running by the way.

btallman · 2016-10-13T04:59:54Z

OK, this is passing CI now. Ready for you to look at it @r39132.

boristyukin · 2016-10-13T16:27:54Z

voting for this PR! IMHO it will make Airflow behave more like enterprise workflow management tools

bolkedebruin · 2016-10-19T13:16:21Z

I would expect this to be part of class DagRun not of TaskInstance.

I put it here, because ti.previous_dagrun() seems cleaner than dr.previous_dagrun(ti)

It is a helper function, as we could also leverage something like:

tip = ti.previous_ti() if tip: drp = tip.get_dagrun()

Dr = Ti.get_dagrun()
Previous = dr.previous_dagrun()

DagRuns are the basis where to work from. TIs should only handle stuff within their own unit. Using previous here disregards that.

OK, cleaned it up... Good idea. Interestingly, there was no dr.get_previous_dagrun, so I added that.

A decision on style for ti.previous_ti vs ti.get_previous_ti() is needed as well, but that isn't something that I introduced.

btallman · 2016-10-20T00:54:26Z

BTW, while it says fail here, all 6 are succeeding on my fork, although I did have to kick one to run again...

r39132 · 2016-10-27T07:07:01Z

@btallman Please rebase. I will be happy to test once ready.

btallman · 2016-10-27T15:50:24Z

@r39132 I went ahead and rebased...

bolkedebruin · 2016-11-13T22:13:08Z

@r39132 did you test this yet? @btallman can you fix the small landscape error? Will schedule some time this week to also tests this.

btallman · 2016-11-13T22:15:47Z

What is the landscape error? Btw, we are running this on half of our production...

Get Outlook for iOS

bolkedebruin · 2016-11-13T22:18:35Z

@btallman one of the comments is too long (click on details for landscape). Good to hear you are running this in production. Would be nice to know a little bit about the complexity of your dags (ie. @r39132 earlier caught some errors that might have slipped through in a not too complex setup).

ldct · 2016-11-19T20:12:08Z

this should be named six_hours_ago_to_the_hour

@zodiac - Missed that one when we switched to testing a longer period. Good catch.

Agree with @zodiac

ldct · 2016-11-19T20:17:00Z

These can be named three_minutes_ago and two_hours_and_three_minutes_ago for consistency

@zodiac - That would be better, not sure if this will make it into a PR.

Agree with @zodiac

mistercrunch · 2016-12-06T17:13:58Z

This overload new meaning into the term backfill. Let's clarify some of the semantics here.

backfill: "a manually triggered CLI process to run or re-run task instances in the past"
scheduler catchup (new term): when the scheduler has task instances to run in the past. This assumes a range of dates between tasks' start_date and the current date, and active DagRuns in between.

I'd suggest renaming DAG.backfill to allow_scheduler_catchup.

mistercrunch · 2016-12-06T17:17:15Z

allow_scheduler_catchup=configuration.getboolean('scheduler', 'allow_scheduler_catchup_default')

mistercrunch · 2016-12-06T17:18:07Z

clearly describes how it affects the scheduler behavihor around DagRun creation

mistercrunch · 2016-12-06T17:20:30Z

# Default setting for `DAG.allow_scheduler_catchup` [clearly describe how it affects DagRun creation] allow_scheduler_catchup_default = True

bolkedebruin · 2016-12-08T20:22:06Z

@mistercrunch I would suggest using DAG.catch_up (or DAG.allow_catch_up) (we are reasoning from the perspective DAG not from the scheduler). In the config section I would suggest to use catch_up_by_default as it already resides under the scheduler section. Bit semantic, but it made it easier in my mind.

r39132 · 2016-12-08T23:29:53Z

@btallman it looks like there is a lot of interest in this PR.. have a look at the comments above and I can help shepherd this through.

bolkedebruin · 2016-12-10T09:19:28Z

This was added in a recent PR, please don't remove it

Yeah, that was a rebase error!!! Sorry.

bolkedebruin · 2016-12-10T09:19:41Z

Idem as below.

bolkedebruin · 2016-12-10T09:20:50Z

While it is outside the scope of this PR, this is potentially a quite expensive operation on the DB.

bolkedebruin · 2016-12-10T09:21:17Z

Please leave description in. See below.

bolkedebruin · 2016-12-10T09:21:32Z

Much cleaner. Thanks!

Going to have to dig into this. Apparently, moving to a DR based previous_ti causes issues when NOT using a DR (backfill). Finally figure that out this afternoon. Will attempt to fix tonight.

bolkedebruin · 2016-12-10T09:24:02Z

Look much better! Just the final dots on I and then we're there. Thanks for all the work and bearing with us.

btallman · 2016-12-10T16:32:01Z

Yeah. Have a bit of cleaning still to do. Thanks, Ben

…

-- Ben Tallman | Senior Technologist - Google Cloud | tallmanb@google.com | 503-680-5709 Sent with help from my rotary phone... On Sat, Dec 10, 2016 at 1:24 AM -0800, "bolkedebruin" <notifications@github.com> wrote: Look much better! Just the final dots on I and then we're there. Thanks for all the work and bearing with us. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

bolkedebruin · 2016-12-25T13:56:39Z

Hey @btallman would be nice to have this in before the new year. Can you rebase and see why the builds are failing? If needed I can provide some help.

btallman · 2016-12-27T02:09:05Z

I will revisit in the morning. Sorry about the delayed update. Thanks, Ben

…

-- Ben Tallman | Senior Technologist - Google Cloud | tallmanb@google.com | 503-680-5709 Sent with help from my rotary phone... On Sun, Dec 25, 2016 at 5:56 AM -0800, "bolkedebruin" <notifications@github.com> wrote: Hey @btallman would be nice to have this in before the new year. Can you rebase and see why the builds are failing? If needed I can provide some help. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

btallman · 2016-12-28T06:02:16Z

Rebased and running tests on a side branch on my fork now. When they pass, I'll push to the tracked branch. Thanks, Ben

…

-- Ben Tallman - 503.680.5709

On Mon, Dec 26, 2016 at 6:08 PM, Ben Tallman ***@***.***> wrote: I will revisit in the morning. Sorry about the delayed update. Thanks, Ben -- Ben Tallman | Senior Technologist - Google Cloud | ***@***.*** | 503-680-5709 <(503)%20680-5709> Sent with help from my rotary phone... On Sun, Dec 25, 2016 at 5:56 AM -0800, "bolkedebruin" < ***@***.***> wrote: Hey @btallman <https://github.com/btallman> would be nice to have this in > before the new year. Can you rebase and see why the builds are failing? If > needed I can provide some help. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#1830 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAKnfvbsxIzFj11ljZ5E5uxcXgQtug_Rks5rLnYdgaJpZM4KRoW8> > . >

btallman · 2016-12-28T08:50:16Z

Seeing errors about api-test stuff. Will investigate tomorrow. Thanks, Ben

…

-- Ben Tallman - 503.680.5709

On Tue, Dec 27, 2016 at 10:01 PM, Ben Tallman ***@***.***> wrote: Rebased and running tests on a side branch on my fork now. When they pass, I'll push to the tracked branch. Thanks, Ben -- Ben Tallman - 503.680.5709 <(503)%20680-5709> On Mon, Dec 26, 2016 at 6:08 PM, Ben Tallman ***@***.***> wrote: > I will revisit in the morning. Sorry about the delayed update. > > Thanks, > Ben > > -- > > Ben Tallman | Senior Technologist - Google Cloud | ***@***.*** | > 503-680-5709 <(503)%20680-5709> > > Sent with help from my rotary phone... > > > > > On Sun, Dec 25, 2016 at 5:56 AM -0800, "bolkedebruin" < > ***@***.***> wrote: > > Hey @btallman <https://github.com/btallman> would be nice to have this >> in before the new year. Can you rebase and see why the builds are failing? >> If needed I can provide some help. >> >> — >> You are receiving this because you were mentioned. >> Reply to this email directly, view it on GitHub >> <#1830 (comment)>, >> or mute the thread >> <https://github.com/notifications/unsubscribe-auth/AAKnfvbsxIzFj11ljZ5E5uxcXgQtug_Rks5rLnYdgaJpZM4KRoW8> >> . >> >

btallman · 2017-01-04T05:11:15Z

@bolkedebruin and @r39132 - I finally was able to figure out what the issue was (turns out using DagRun to get previous TI only works if there IS a dag run (so not on backfill, go figure) so that minor change that I agreed to for Bolke turned out to be an issue.

Tracked that down and the tests are now passing.

Please review ASAP.

bolkedebruin · 2017-01-04T16:05:37Z

@btallman I need to look in to your comment on backfills, as in master backfills actually create dag runs. Can you elaborate a bit further what you think went wrong?

btallman · 2017-01-04T21:12:58Z

It may not actually be backfill, it's just that at least one of the tests generates a taskinstance from dag.tasks[0] dynamically and then runs the task in which case there's actually no dagrun. I trapped for that and it works as expected. Possibly the test should be rewritten instead. Thanks, Ben

…

-- Ben Tallman | Senior Technologist - Google Cloud | tallmanb@google.com | 503-680-5709 Sent with help from my rotary phone... On Wed, Jan 4, 2017 at 8:05 AM -0800, "bolkedebruin" <notifications@github.com> wrote: @btallman I need to look in to your comment on backfills, as in master backfills actually create dag runs. Can you elaborate a bit further what you think went wrong? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

bolkedebruin · 2017-01-04T21:15:11Z

Gotcha. That makes sense.

bolkedebruin

Some minor things. I will start integration testing now.

bolkedebruin · 2017-01-06T13:31:32Z

+            DagRun.execution_date.desc()
+        ).first()
+
+    @provide_session


I'm not sure if I understand the difference between get_previous_dagrun and get_previous_scheduled_dagrun . Should get_previous_dagrun also not return the same as get_previous_scheduled_dagrun? Or do you want to distinguish between backfilled and intervalled dag runs?

Please rename to "get_previous_scheduled" it is already part of DagRun.

bolkedebruin · 2017-01-06T13:33:56Z

+        dag = self.task.dag
+        if dag:
+            dr = self.get_dagrun(session=session)
+            if not dr:


I understand why you are doing this, although the comment does not reflect the requirement I think. Please mark it as FIXME (it should be removed in the future / throw an exception)

bolkedebruin · 2017-01-06T13:36:11Z

+                if not previous_scheduled_date:
+                    return None
+                else:
+                    return TaskInstance(task=self.task, execution_date=previous_scheduled_date)


This means that whether or not there is a "previous_scheduled_date" (i.e. there won't be one by convention before start_date) one always gets a TaskInstance. I don't think that should be the case.

bolkedebruin · 2017-01-06T13:36:49Z


        return self.dag

+    @provide_session


please rename to "get_previous", it is already part of DagRun (or even make it a property)

bolkedebruin · 2017-01-06T13:40:14Z

 authenticate = true
 max_threads = 2
+catchup_by_default = True
+scheduler_zombie_task_threshold = 300


This is a duplicate from above

bolkedebruin · 2017-01-06T14:10:40Z

In initial testing it seems to behave quite ok (thats an understatement for currently flawless)

bolkedebruin · 2017-01-06T14:13:41Z

What's left (apart from the minor changes) is to make sure it is documented, maybe with a small example.

btallman · 2017-01-06T22:49:24Z

Yes, I had it documented, but that fell out when I moved it to catchup... Totally forgot it. Thanks, Ben

…

-- Ben Tallman - 503.680.5709

On Fri, Jan 6, 2017 at 6:13 AM, bolkedebruin ***@***.***> wrote: What's left (apart from the minor changes) is to make sure it is documented, maybe with a small example. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1830 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKnfvyZtgiaEzhBeDjyqsXh03sGHN3Oks5rPkwbgaJpZM4KRoW8> .

bolkedebruin · 2017-01-11T07:53:48Z

Hey @btallman can you add the documentation, I would really like to merge this and get it into 1.8. (And the minor nits)

btallman · 2017-01-11T23:16:05Z

Done. Thanks, Ben

…

-- Ben Tallman - 503.680.5709

On Tue, Jan 10, 2017 at 11:53 PM, bolkedebruin ***@***.***> wrote: Hey @btallman <https://github.com/btallman> can you add the documentation, I would really like to merge this and get it into 1.8. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1830 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKnflyk1p7cVbLJVMQ3t7kZkVeGLL0tks5rRIqVgaJpZM4KRoW8> .

btallman · 2017-01-11T23:24:04Z

BTW - Although this failed tests on here, it succeeded on my fork... I'll re-push a NON update to force a re-test I guess. - https://travis-ci.org/btallman/incubator-airflow

Added a dag.catchup option and modified the scheduler to look at the value when scheduling DagRuns (by moving dag.start_date up to dag.previous_schedule), and added a config option catchup_by_default (defaults to True) that allows users to set this to False for all dags modifying the existing DAGs In addition, we added a test to jobs.py (test_dag_catchup_option)

bolkedebruin · 2017-01-13T11:38:04Z

That seems to be an error in an assumption on one of the tests. Tests were executed at 23 hours. The test was doing now.hour() + 1, while hours should stay within 0..23 .

Will merge. Thanks a lot @btallman !

Added a dag.catchup option and modified the scheduler to look at the value when scheduling DagRuns (by moving dag.start_date up to dag.previous_schedule), and added a config option catchup_by_default (defaults to True) that allows users to set this to False for all dags modifying the existing DAGs In addition, we added a test to jobs.py (test_dag_catchup_option) Closes apache#1830 from btallman/NoBackfill_clean_feature

bolkedebruin reviewed Oct 9, 2016

View reviewed changes

btallman force-pushed the NoBackfill_clean_feature branch 3 times, most recently from 2197ab1 to 9fe4f83 Compare October 13, 2016 04:35

bolkedebruin requested changes Oct 19, 2016

View reviewed changes

btallman force-pushed the NoBackfill_clean_feature branch 2 times, most recently from 4714683 to 4ac201d Compare October 19, 2016 21:01

btallman force-pushed the NoBackfill_clean_feature branch from 4ac201d to f758c01 Compare October 27, 2016 15:49

btallman force-pushed the NoBackfill_clean_feature branch 2 times, most recently from 567076d to 6c87204 Compare November 15, 2016 06:31

ldct reviewed Nov 19, 2016

View reviewed changes

mistercrunch requested changes Dec 6, 2016

View reviewed changes

btallman force-pushed the NoBackfill_clean_feature branch from 99b67f9 to 754d96f Compare December 8, 2016 21:56

bolkedebruin reviewed Dec 10, 2016

View reviewed changes

btallman force-pushed the NoBackfill_clean_feature branch from 754d96f to f296939 Compare January 4, 2017 05:07

bolkedebruin requested changes Jan 6, 2017

View reviewed changes

btallman force-pushed the NoBackfill_clean_feature branch from f296939 to de0c044 Compare January 11, 2017 22:41

btallman force-pushed the NoBackfill_clean_feature branch from de0c044 to d2a5f46 Compare January 11, 2017 23:25

asfgit closed this in 1caaceb Jan 13, 2017

Conversation

btallman commented Oct 8, 2016

Uh oh!

bolkedebruin commented Oct 8, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

btallman commented Oct 8, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

btallman commented Oct 9, 2016

Uh oh!

codecov-io commented Oct 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Current coverage is 67.01% (diff: 67.39%)

Uh oh!

r39132 commented Oct 9, 2016

Uh oh!

btallman commented Oct 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

r39132 commented Oct 10, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

r39132 commented Oct 10, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

btallman commented Oct 11, 2016

Uh oh!

r39132 commented Oct 13, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

btallman commented Oct 13, 2016

Uh oh!

boristyukin commented Oct 13, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

btallman commented Oct 20, 2016

Uh oh!

r39132 commented Oct 27, 2016

Uh oh!

btallman commented Oct 27, 2016

Uh oh!

bolkedebruin commented Nov 13, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

btallman commented Nov 13, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bolkedebruin commented Nov 13, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mistercrunch commented Dec 6, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

bolkedebruin commented Oct 8, 2016 •

edited

Loading

btallman commented Oct 8, 2016 •

edited

Loading

codecov-io commented Oct 9, 2016 •

edited

Loading

btallman commented Oct 9, 2016 •

edited

Loading

r39132 commented Oct 10, 2016 •

edited

Loading

r39132 commented Oct 10, 2016 •

edited

Loading

r39132 commented Oct 13, 2016 •

edited

Loading

bolkedebruin commented Nov 13, 2016 •

edited

Loading

btallman commented Nov 13, 2016 •

edited

Loading