Add benchmark testing apis

[ChengFR]

The following tasks have been accomplished:

• add benchmark testing APIs in cardea/benchmark (Resolve Benchmark testing apis #64).
• create customized primitives in cardea/primitives and makes the setup (Resolve Primitive setup #61).
• add a function execute_pipeline_from_pipeline in cardea/modeling/modeler.py. This function loads MLPipeline instances instead of primitives for testing (Resolve Modeler load pipelines instead of lists of primitives #65).

[CLAassistant]

All committers have signed the CLA.

[sarahmish]

Looks great @ChengFR! I have some minor comments, if you would kindly address them.

I also want to leave out the python notebooks that are on the other branch (1, 2, 3, and 4) from this PR, and keep it dedicated to benchmarking (benchmarking and tasks can be left for illustration usage purposes).

[sarahmish]

Load/ Construct Dataset. In the future, the "data_loader" and "problem_definition" will construct the feature matrix. I am thinking we can make feature_matrix a more generic name like, dataset. which can be used for all stages to generate the feature matrix.

Dataset can be:

• feature matrix (pandas.DataFrame).
• entityset (Featuretools.EntitySet).
• raw data (list of pandas.DataFrame).

This is different than the consensus we reached in the design document, but is more generic.. Otherwise always assume I am reading my data from a path_to_dataset from the task itself.

We can proceed without doing any changes, but I am just marking them here in case we refer back to this idea.

[ChengFR]

The feature_matrix in the inputs of evaluate_task is a reserved parameter. When we run multiple pipelines on the same dataset from an entityset or raw data, we don't want to run the featurization multiple times. In this case, the upper-level function benchmark will do the featurization and pass the feature_matrix to evaluate_task.

So I don't think it is necessary to make it more general.

[sarahmish]

Is this necessary? I tried running the pipeline directly and it was fine.

Our definition of MLBLOCKS_PRIMITIVES is in the entry point (view setup.py), so it should be working.

[ChengFR]

It seems that in my testing environment, without this sentence, the customized primitives cannot be found. Can you double-check this problem?

[sarahmish]

After investigating this, the entry_point mentioned is only possible in the newer version of MLBlocks. We can keep this implementation for now, but mark it to be removed after the upgrade.

[sarahmish]

I would use another approach (same as the reason earlier).

from import dirname

PIPELINE_DIR  = os.path.join(os.path.dirname(__file__), 'pipelines')
VERIFIED_DIR = os.path.join(os.path.dirname(os.path.dirname(__file__)), 'benchmark', 'verified')

[ChengFR]

Current paths in task configurations are all relative paths from the root. Since we will share the benchmarking results, this design ensures that the tasks are runnable on others' devices. Are you suggesting we use absolute paths instead (which is more readable but not runnable on different devices)?

[sarahmish]

I see your intention but the problem is this is difficult to trace. My suggestion would be to have the absolute path and the users should be following the same configuration of Cardea.

I imagine that PIPELINE_DIR will be defined in the same manner (in all cases) but the VERIFIED_DIR will change depending on the user. Is this the case you are referring to by

not runnable on different devices?

If that's what you mean, then I think that is not an issue, it can always be overriden by the user to specify their own path. But generally speaking, absolute paths are more maintainable and traceable as well.

[sarahmish]

Looks good @ChengFR!