Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More descriptive DFS progress bar #69

Merged
merged 13 commits into from Mar 19, 2018
Merged

More descriptive DFS progress bar #69

merged 13 commits into from Mar 19, 2018

Conversation

bschreck
Copy link
Contributor

Updated DFS progress bar to display number of features actually constructed, as well features removed at the end due to restrictions on variable types and the drop_contains/drop_exact list.

new_features = self._filter_features(new_features)
if verbose and before_len > len(new_features):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does this look like when it gets written?

I like the main change, but not sure it's that helpful to have this print out as things are removed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It actually looks pretty good I think, try running it and check it out.

Basically the progress bar is like:

[45 features] 100 features/s

Eventually the feature building stops and the progress bar stops, then above the progress bar the number removed is written:

Removed 2 features with undesired variable types
Removed 2 features from drop_contains list
[45 features] 100 features/s

@codecov-io
Copy link

codecov-io commented Jan 16, 2018

Codecov Report

Merging #69 into master will increase coverage by 0.02%.
The diff coverage is 33.33%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #69      +/-   ##
==========================================
+ Coverage    88.3%   88.33%   +0.02%     
==========================================
  Files          73       73              
  Lines        7447     7439       -8     
==========================================
- Hits         6576     6571       -5     
+ Misses        871      868       -3
Impacted Files Coverage Δ
featuretools/synthesis/deep_feature_synthesis.py 93.06% <0%> (+0.95%) ⬆️
...computational_backends/calculate_feature_matrix.py 96.88% <100%> (+0.01%) ⬆️
featuretools/utils/gen_utils.py 64.44% <0%> (-2.23%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 959069d...d219a0a. Read the comment docs.

@Seth-Rothschild
Copy link
Contributor

Here's a screenshot.
image
The top three lines are new and give some information about what DFS has done. I think that's probably a step in the right direction.

It might be worth modifying some of the words around the pbar itself through tqdm. The line

1/1 [00:03<00:00,  3.16s/cutoff time]

probably doesn't give enough information to a new user. It presupposes they know that the 1/1 is counting the cutoff times they've given, and that they don't parse < to mean "elapsed time is less than remaining time".

How about something like

Elapsed: 1:30 | Remaining 4:10 |+-bar-here-----| 
Calculated matrix at 3/12 cutoff times (avg. 30s/calc): 

@bschreck
Copy link
Contributor Author

bschreck commented Feb 7, 2018

Yup that all makes sense. I didn't test the actual feature calculation step, just the building step (because I noticed what we show for that step right now doesn't make much sense).

@Seth-Rothschild
Copy link
Contributor

Seems difficult to get multiple lines into the bar_format in tqdm. Here's a single line version
image

@bschreck
Copy link
Contributor Author

Updated to include feature exclusion line inside of pbar:
pbar

@bschreck
Copy link
Contributor Author

Now looks like this:
pbar

@Seth-Rothschild
Copy link
Contributor

I'm really enjoying using this.

A small note is that the order/phrasing of the drop contains message is a little confusing:
image
Here there are 18 (12 dropped + 6 excluded) features which aren't calculated. Is it possible to add the 12 into features_excluded or to move the "Removed 12 features..." line to after the calculation?

@bschreck
Copy link
Contributor Author

I didn't see that line- should be fixed now

@bschreck
Copy link
Contributor Author

Now with removed pbar from Deep Feature Synthesis.

In progress:
in_progress_pbar

Complete:
done_pbar

@kmax12
Copy link
Contributor

kmax12 commented Mar 19, 2018

Looks great!

@kmax12 kmax12 merged commit af88087 into master Mar 19, 2018
@rwedge rwedge mentioned this pull request Mar 21, 2018
@Seth-Rothschild Seth-Rothschild deleted the better-dfs-pbar branch August 10, 2018 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants