New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More descriptive DFS progress bar #69

Merged
merged 13 commits into from Mar 19, 2018

Conversation

Projects
None yet
4 participants
@bschreck
Contributor

bschreck commented Jan 16, 2018

Updated DFS progress bar to display number of features actually constructed, as well features removed at the end due to restrictions on variable types and the drop_contains/drop_exact list.

new_features = self._filter_features(new_features)
if verbose and before_len > len(new_features):

This comment has been minimized.

@kmax12

kmax12 Jan 16, 2018

Member

what does this look like when it gets written?

I like the main change, but not sure it's that helpful to have this print out as things are removed.

This comment has been minimized.

@bschreck

bschreck Jan 17, 2018

Contributor

It actually looks pretty good I think, try running it and check it out.

Basically the progress bar is like:

[45 features] 100 features/s

Eventually the feature building stops and the progress bar stops, then above the progress bar the number removed is written:

Removed 2 features with undesired variable types
Removed 2 features from drop_contains list
[45 features] 100 features/s

@codecov-io

This comment has been minimized.

codecov-io commented Jan 16, 2018

Codecov Report

Merging #69 into master will increase coverage by 0.02%.
The diff coverage is 33.33%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #69      +/-   ##
==========================================
+ Coverage    88.3%   88.33%   +0.02%     
==========================================
  Files          73       73              
  Lines        7447     7439       -8     
==========================================
- Hits         6576     6571       -5     
+ Misses        871      868       -3
Impacted Files Coverage Δ
featuretools/synthesis/deep_feature_synthesis.py 93.06% <0%> (+0.95%) ⬆️
...computational_backends/calculate_feature_matrix.py 96.88% <100%> (+0.01%) ⬆️
featuretools/utils/gen_utils.py 64.44% <0%> (-2.23%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 959069d...d219a0a. Read the comment docs.

@Seth-Rothschild

This comment has been minimized.

Contributor

Seth-Rothschild commented Jan 20, 2018

Here's a screenshot.
image
The top three lines are new and give some information about what DFS has done. I think that's probably a step in the right direction.

It might be worth modifying some of the words around the pbar itself through tqdm. The line

1/1 [00:03<00:00,  3.16s/cutoff time]

probably doesn't give enough information to a new user. It presupposes they know that the 1/1 is counting the cutoff times they've given, and that they don't parse < to mean "elapsed time is less than remaining time".

How about something like

Elapsed: 1:30 | Remaining 4:10 |+-bar-here-----| 
Calculated matrix at 3/12 cutoff times (avg. 30s/calc): 
@bschreck

This comment has been minimized.

Contributor

bschreck commented Feb 7, 2018

Yup that all makes sense. I didn't test the actual feature calculation step, just the building step (because I noticed what we show for that step right now doesn't make much sense).

@Seth-Rothschild Seth-Rothschild force-pushed the better-dfs-pbar branch from 7aea72f to 84316cd Feb 7, 2018

@Seth-Rothschild

This comment has been minimized.

Contributor

Seth-Rothschild commented Feb 7, 2018

Seems difficult to get multiple lines into the bar_format in tqdm. Here's a single line version
image

@bschreck

This comment has been minimized.

Contributor

bschreck commented Mar 15, 2018

Updated to include feature exclusion line inside of pbar:
pbar

bschreck added some commits Mar 15, 2018

@bschreck

This comment has been minimized.

Contributor

bschreck commented Mar 15, 2018

Now looks like this:
pbar

@Seth-Rothschild

This comment has been minimized.

Contributor

Seth-Rothschild commented Mar 16, 2018

I'm really enjoying using this.

A small note is that the order/phrasing of the drop contains message is a little confusing:
image
Here there are 18 (12 dropped + 6 excluded) features which aren't calculated. Is it possible to add the 12 into features_excluded or to move the "Removed 12 features..." line to after the calculation?

@bschreck

This comment has been minimized.

Contributor

bschreck commented Mar 16, 2018

I didn't see that line- should be fixed now

@bschreck

This comment has been minimized.

Contributor

bschreck commented Mar 19, 2018

Now with removed pbar from Deep Feature Synthesis.

In progress:
in_progress_pbar

Complete:
done_pbar

@kmax12

This comment has been minimized.

Member

kmax12 commented Mar 19, 2018

Looks great!

@kmax12 kmax12 merged commit af88087 into master Mar 19, 2018

2 checks passed

ci/circleci Your tests passed on CircleCI!
Details
license/cla Contributor License Agreement is signed.
Details

@rwedge rwedge referenced this pull request Mar 21, 2018

Merged

Release v0.1.19 #114

@Seth-Rothschild Seth-Rothschild deleted the better-dfs-pbar branch Aug 10, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment