Skip to content

[query/service] make available driver progress information #11746

Merged
danking merged 11 commits into
hail-is:mainfrom
danking:sb-client-knows-driver
May 13, 2022
Merged

[query/service] make available driver progress information #11746
danking merged 11 commits into
hail-is:mainfrom
danking:sb-client-knows-driver

Conversation

@danking

@danking danking commented Apr 8, 2022

Copy link
Copy Markdown
Contributor

It now looks like this:

In [2]: import hail as hl
   ...: t = hl.utils.range_table(100)
   ...: t._force_count()
   ...: t.show()
Submitted batch 1774275, see https://batch.hail.is/batches/1774275/jobs/1
Batch 1774275: completed jobs: 100%|████████████████████████████████████████████████████████████████████| 1/1 [00:14<00:00, 14.76s/it]
Submitted batch 1774315, see https://batch.hail.is/batches/1774315/jobs/1
Batch 1774315: completed jobs: 100%|████████████████████████████████████████████████████████████████████| 1/1 [01:03<00:00, 63.06s/it]
+-------+
|   idx |
+-------+
| int32 |
+-------+
|     0 |
|     1 |
|     2 |
|     3 |
|     4 |
|     5 |
|     6 |
|     7 |
|     8 |
|     9 |
|    10 |
|    11 |
|    12 |
|    13 |
+-------+
showing top 14 rows

Daniel King added 8 commits April 1, 2022 15:42
These changes make it straightforward to always use QoB by default.
It now looks like this:
```
In [2]: import hail as hl
   ...: t = hl.utils.range_table(100)
   ...: t._force_count()
   ...: t.show()
Submitted batch 1774275, see https://batch.hail.is/batches/1774275/jobs/1
Batch 1774275: completed jobs: 100%|████████████████████████████████████████████████████████████████████| 1/1 [00:14<00:00, 14.76s/it]
Submitted batch 1774315, see https://batch.hail.is/batches/1774315/jobs/1
Batch 1774315: completed jobs: 100%|████████████████████████████████████████████████████████████████████| 1/1 [01:03<00:00, 63.06s/it]
+-------+
|   idx |
+-------+
| int32 |
+-------+
|     0 |
|     1 |
|     2 |
|     3 |
|     4 |
|     5 |
|     6 |
|     7 |
|     8 |
|     9 |
|    10 |
|    11 |
|    12 |
|    13 |
+-------+
showing top 14 rows
```
@danking

danking commented Apr 8, 2022

Copy link
Copy Markdown
Contributor Author

stacked on #11745

@danking danking assigned tpoterba and unassigned jigold Apr 20, 2022
@danking

danking commented Apr 20, 2022

Copy link
Copy Markdown
Contributor Author

@tpoterba I have two concrete questions for you here:

  1. What do you think of this (very minimal) amount of information about the driver batch?
  2. What should we do about boolean fields going forward? Here I only accept the strings "True" and "true" but I think we've thus far accepted any non-empty string as a true value. I suppose we should probably continue that process, but I'd like to get your thoughts before I go that route.

@tpoterba

Copy link
Copy Markdown
Contributor
  1. I think it's better than nothing! Ship it.

  2. This is for env vars, right? I think people often use HAIL_BLAH=1 from the command line to set stuff, so maybe look for that too? Chris probably has more grounded opinions about this.

@tpoterba

Copy link
Copy Markdown
Contributor

@chrisvittal can you comment on this second point?

@chrisvittal

Copy link
Copy Markdown
Collaborator

For flag style env variables, I think we should use present and non-empty. We could also go with a reasonable set of 'affirmative' values, like say (all case insensitive) true, t, yes, y, 1.

@danking

danking commented Apr 20, 2022

Copy link
Copy Markdown
Contributor Author

I'll go with present and non-empty.

@danking

danking commented Apr 20, 2022

Copy link
Copy Markdown
Contributor Author

OK, ready for review.

@tpoterba tpoterba left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, didn't realize this was hanging on me!

@danking
danking merged commit b4feb52 into hail-is:main May 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants