Refactor deploy_to_develop pipeline in tech-specific pipelines

[pombredanne]

We want to be able to run d2d combined and separately:

• Create d2d analyze_deployed_java pipeline
• Create d2d analyze_deployed_javascript pipeline
• Create d2d match_code pipeline

[tdruez]

Here's the list of steps for the current d2d pipeline:

• get_inputs
• extract_inputs_to_codebase_directory
• extract_archives_in_place
• collect_and_create_codebase_resources
• fingerprint_codebase_directories
• flag_empty_files
• flag_whitespace_files
• flag_ignored_resources
• map_about_files
• map_checksum
• match_archives_to_purldb
• find_java_packages
• map_java_to_class
• map_jar_to_source
• map_javascript
• match_directories_to_purldb
• match_resources_to_purldb
• map_javascript_post_purldb_match
• map_javascript_path
• map_javascript_colocation
• map_thirdparty_npm_packages
• map_path
• flag_mapped_resources_archives_and_ignored_directories
• perform_house_keeping_tasks
• match_purldb_resources_post_process
• remove_packages_without_resources
• scan_unmapped_to_files
• scan_mapped_from_for_files
• flag_deployed_from_resources_with_missing_license
• create_local_files_packages

The Java-specific steps are:

• find_java_packages
• map_java_to_class
• map_jar_to_source

The JavaScript-specific steps are:

• map_javascript
• map_javascript_post_purldb_match. (warning dependency on the match_to_purldb steps)
• map_javascript_path
• map_javascript_colocation
• map_thirdparty_npm_packages

Now in case we want to have a tech-specific only type of d2d pipeline, how do we want to deal with the 20+ other steps that are not tech-specific? ie: extract, collect, fingerprint, flag, map about, map checksum, match purldb, cleanup, scan, etc...

If we duplicate those steps across all d2d_* pipeline, how can we run multiple of those on the same project?

---

An alternative idea would be to decorate/flag some steps as technology-specific, and define/enable on the project (or pipeline args) the tech to be handle in the single d2d pipeline.

class DeployToDevelop(Pipeline):
 [...]
 
 @tag("java")
 def find_java_packages(self):
     """Find the java package of the .java source files."""
     d2d.find_java_packages(self.project, logger=self.log)

 @tag("javascript", "purldb")
 def map_javascript_post_purldb_match(self):
     """Map minified javascript file based on existing PurlDB match."""
     d2d.map_javascript_post_purldb_match(project=self.project, logger=self.log)

pipeline = DeployToDevelop()
pipeline.enabled_tags = ["java", "purldb"]
pipeline.save()

Available tags/flags would be available in the UI as checkboxes:

• Java
• JavaScript
• Python
• C

Flagged steps that are not enabled will be skipped during the pipeline execution.
This implementation would not be d2d specific but used globally as a way to easily exclude steps from pipeline execution.

@pombredanne Let me know your take on this approach.

[pombredanne]

@tdruez I like the approach. We could even refine this later to have an "automatic" feature that would guess the technologies.
We need to have it selectable fopr now though, so you approach makes sense

[tdruez]

Implemented in #1055