Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[query/service] make available driver progress information #11746

Merged
merged 11 commits into from
May 13, 2022

Conversation

danking
Copy link
Contributor

@danking danking commented Apr 8, 2022

It now looks like this:

In [2]: import hail as hl
   ...: t = hl.utils.range_table(100)
   ...: t._force_count()
   ...: t.show()
Submitted batch 1774275, see https://batch.hail.is/batches/1774275/jobs/1
Batch 1774275: completed jobs: 100%|████████████████████████████████████████████████████████████████████| 1/1 [00:14<00:00, 14.76s/it]
Submitted batch 1774315, see https://batch.hail.is/batches/1774315/jobs/1
Batch 1774315: completed jobs: 100%|████████████████████████████████████████████████████████████████████| 1/1 [01:03<00:00, 63.06s/it]
+-------+
|   idx |
+-------+
| int32 |
+-------+
|     0 |
|     1 |
|     2 |
|     3 |
|     4 |
|     5 |
|     6 |
|     7 |
|     8 |
|     9 |
|    10 |
|    11 |
|    12 |
|    13 |
+-------+
showing top 14 rows

Daniel King added 8 commits April 1, 2022 15:42
These changes make it straightforward to always use QoB by default.
It now looks like this:
```
In [2]: import hail as hl
   ...: t = hl.utils.range_table(100)
   ...: t._force_count()
   ...: t.show()
Submitted batch 1774275, see https://batch.hail.is/batches/1774275/jobs/1
Batch 1774275: completed jobs: 100%|████████████████████████████████████████████████████████████████████| 1/1 [00:14<00:00, 14.76s/it]
Submitted batch 1774315, see https://batch.hail.is/batches/1774315/jobs/1
Batch 1774315: completed jobs: 100%|████████████████████████████████████████████████████████████████████| 1/1 [01:03<00:00, 63.06s/it]
+-------+
|   idx |
+-------+
| int32 |
+-------+
|     0 |
|     1 |
|     2 |
|     3 |
|     4 |
|     5 |
|     6 |
|     7 |
|     8 |
|     9 |
|    10 |
|    11 |
|    12 |
|    13 |
+-------+
showing top 14 rows
```
@danking
Copy link
Contributor Author

danking commented Apr 8, 2022

stacked on #11745

@danking danking assigned tpoterba and unassigned jigold Apr 20, 2022
@danking
Copy link
Contributor Author

danking commented Apr 20, 2022

@tpoterba I have two concrete questions for you here:

  1. What do you think of this (very minimal) amount of information about the driver batch?
  2. What should we do about boolean fields going forward? Here I only accept the strings "True" and "true" but I think we've thus far accepted any non-empty string as a true value. I suppose we should probably continue that process, but I'd like to get your thoughts before I go that route.

@tpoterba
Copy link
Contributor

  1. I think it's better than nothing! Ship it.

  2. This is for env vars, right? I think people often use HAIL_BLAH=1 from the command line to set stuff, so maybe look for that too? Chris probably has more grounded opinions about this.

@tpoterba
Copy link
Contributor

@chrisvittal can you comment on this second point?

@chrisvittal
Copy link
Collaborator

For flag style env variables, I think we should use present and non-empty. We could also go with a reasonable set of 'affirmative' values, like say (all case insensitive) true, t, yes, y, 1.

@danking
Copy link
Contributor Author

danking commented Apr 20, 2022

I'll go with present and non-empty.

@danking
Copy link
Contributor Author

danking commented Apr 20, 2022

OK, ready for review.

Copy link
Contributor

@tpoterba tpoterba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, didn't realize this was hanging on me!

@danking danking merged commit b4feb52 into hail-is:main May 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants