Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Review Oracle supporting objects #18

Closed
nj1973 opened this issue Oct 13, 2023 · 4 comments · Fixed by #51
Closed

Review Oracle supporting objects #18

nj1973 opened this issue Oct 13, 2023 · 4 comments · Fixed by #51
Assignees
Labels
framework Setup tasks relating to builds, installation and dev initial release Required for initial open source effort

Comments

@nj1973
Copy link
Collaborator

nj1973 commented Oct 13, 2023

We've already removed a lot of PL/SQL code from the ADM schema but should conduct a further review and remove anything we no longer require.

Other thoughts:

  • Shall we change the default account prefix from "GLUENT" to "GOE"?
  • Now is the right time to remove any metadata attributes that have become redundant. This is a fresh start so no upgrades are required.
  • We should review the repo schema table structures. For example should metadata now be keyed on the source table and no longer on the hybrid view? I can foresee a future where Offload needs to reinstate the ability to update a HWM in a dependent view but I think these would be recorded in a new child table
  • We can reset the repo schema version to an initial version, we do not need to honour upgrades from existing GDP installations
@nj1973 nj1973 added framework Setup tasks relating to builds, installation and dev initial release Required for initial open source effort labels Oct 13, 2023
@abb9979 abb9979 self-assigned this Nov 23, 2023
@abb9979
Copy link
Contributor

abb9979 commented Nov 23, 2023

Renaming GLUENT to GOE throughout the SQL code.

@abb9979
Copy link
Contributor

abb9979 commented Nov 23, 2023

Audit of current COMMAND_STEP usage mid-cleanup:

ALTER_TABLE
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_ALTER_TABLE = 'ALTER_TABLE'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_ALTER_TABLE: 'Alter backend table',
./gluentlib/gluentlib/offload/backend_table.py:        return self._offload_step(command_steps.STEP_ALTER_TABLE, step_fn)
==============================================================================
ANALYZE_DATA_TYPES
./scripts/gluent.py:        messages.offload_step(command_steps.STEP_ANALYZE_DATA_TYPES,
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_ANALYZE_DATA_TYPES = 'ANALYZE_DATA_TYPES'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_ANALYZE_DATA_TYPES: 'Analyzing data types',
==============================================================================
COMPACTION_ADD_DELTA_PARTITION
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_COMPACT_ADD_DELTA_PARTITION = 'COMPACTION_ADD_DELTA_PARTITION'
==============================================================================
COMPACTION_BACKUP_PARTITION
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_COMPACT_BACKUP_PARTITION = 'COMPACTION_BACKUP_PARTITION'
==============================================================================
COMPACTION_BACKUP_TABLE
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_COMPACT_BACKUP_TABLE = 'COMPACTION_BACKUP_TABLE'
==============================================================================
COMPUTE_BASE_STATS
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_COMPACT_COMPUTE_BASE_STATS = 'COMPUTE_BASE_STATS'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_COMPACT_COMPUTE_BASE_STATS: 'Compute stats for new base table',
==============================================================================
COMPUTE_PART_STATS
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_COMPACT_COMPUTE_PART_STATS = 'COMPUTE_PART_STATS'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_COMPACT_COMPUTE_PART_STATS: 'Compute stats for compacted partitions',
==============================================================================
COMPUTE_STATS
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_COMPUTE_STATS = 'COMPUTE_STATS'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_COMPUTE_STATS: 'Compute backend statistics',
./gluentlib/gluentlib/offload/backend_table.py:            self._offload_step(command_steps.STEP_COMPUTE_STATS,
==============================================================================
COPY_EXTERNAL_STATS
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_EXT_STATS_COPY = 'COPY_EXTERNAL_STATS'
==============================================================================
COPY_STATS_TO_BACKEND
./scripts/gluent.py:          command_steps.STEP_COPY_STATS_TO_BACKEND,
./gluentlib/tests/integration/test_sets/stories/test_stories/backend_stats.py:             (lambda test: text_in_warnings(test.offload_messages, step_title(command_steps.STEP_COPY_STATS_TO_BACKEND)),
./gluentlib/tests/integration/test_sets/stories/test_stories/backend_stats.py:             (lambda test: text_in_warnings(test.offload_messages, step_title(command_steps.STEP_COPY_STATS_TO_BACKEND)),
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_COPY_STATS_TO_BACKEND = 'COPY_STATS_TO_BACKEND'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_COPY_STATS_TO_BACKEND: 'Copy RDBMS stats to Backend',
==============================================================================
COUNT_AAPD
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Skipping Step: %s' % step_title(command_steps.STEP_COUNT_AAPD),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                     'skip': [step_title_to_step_id(step_title(command_steps.STEP_COUNT_AAPD))],
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:             (lambda test: messages_step_executions(test.offload_messages, step_title(command_steps.STEP_COUNT_AAPD)),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                                ',' + step_title_to_step_id(step_title(command_steps.STEP_COUNT_AAPD))])],
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_COUNT_AAPD = 'COUNT_AAPD'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_COUNT_AAPD: 'Create count star aggregation rule',
==============================================================================
CREATE_BASE_TABLE
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_COMPACT_CREATE_BASE_TABLE = 'CREATE_BASE_TABLE'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_COMPACT_CREATE_BASE_TABLE: 'Create new base table',
==============================================================================
CREATE_CONV_VIEW
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_CREATE_CONV_VIEW = 'CREATE_CONV_VIEW'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_CREATE_CONV_VIEW: 'Create data-conversion view',
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_CREATE_CONV_VIEW, step_fn,
==============================================================================
CREATE_DB
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_CREATE_DB = 'CREATE_DB'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_CREATE_DB: 'Create backend database',
./gluentlib/gluentlib/offload/offload_constants.py:""" CAPABILITY_CREATE_DB
./gluentlib/gluentlib/offload/offload_constants.py:CAPABILITY_CREATE_DB = 'create_db'
./gluentlib/gluentlib/offload/offload_constants.py:    CAPABILITY_CREATE_DB: True,
./gluentlib/gluentlib/offload/offload_constants.py:    CAPABILITY_CREATE_DB: True,
./gluentlib/gluentlib/offload/offload_constants.py:    CAPABILITY_CREATE_DB: True,
./gluentlib/gluentlib/offload/offload_constants.py:    CAPABILITY_CREATE_DB: True,
./gluentlib/gluentlib/offload/backend_api.py:    CAPABILITY_CREATE_DB, CAPABILITY_DROP_COLUMN, CAPABILITY_FS_SCHEME_ABFS, CAPABILITY_FS_SCHEME_ADL,\
./gluentlib/gluentlib/offload/backend_api.py:        return self.is_capability_supported(CAPABILITY_CREATE_DB)
./gluentlib/gluentlib/offload/backend_table.py:            self._offload_step(command_steps.STEP_CREATE_DB, lambda: self.create_db())
==============================================================================
CREATE_DELTA_TABLE
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_CREATE_DELTA_TABLE = 'CREATE_DELTA_TABLE'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_CREATE_DELTA_TABLE: 'Create delta table',
==============================================================================
CREATE_EXTERNAL_TABLE
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_CREATE_EXTERNAL_TABLE = 'CREATE_EXTERNAL_TABLE'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_CREATE_EXTERNAL_TABLE: 'Create external table',
==============================================================================
CREATE_EXTRACTION_TMP_DIR
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_EXTRACT_TMP_DIR = 'CREATE_EXTRACTION_TMP_DIR'
==============================================================================
CREATE_HDFS_SNAPSHOT
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_COMPACT_HDFS_SNAPSHOT = 'CREATE_HDFS_SNAPSHOT'
==============================================================================
CREATE_HYBRID_VIEW
./gluentlib/tests/integration/persistence/test_orchestration_repo_client.py:            command_steps.STEP_CREATE_HYBRID_VIEW,
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_CREATE_HYBRID_VIEW = 'CREATE_HYBRID_VIEW'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_CREATE_HYBRID_VIEW: 'Create hybrid view',
==============================================================================
CREATE_INTERNAL_VIEW
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_COMPACT_CREATE_INTL_VIEW = 'CREATE_INTERNAL_VIEW'
==============================================================================
CREATE_JOIN_VIEW
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_CREATE_JOIN_VIEW = 'CREATE_JOIN_VIEW'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_CREATE_JOIN_VIEW: 'Create join view',
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_CREATE_JOIN_VIEW, step_fn)
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_CREATE_JOIN_VIEW, step_fn)
==============================================================================
CREATE_REWRITE_RULES
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:             (lambda test: text_in_log(step_title(command_steps.STEP_CREATE_REWRITE_RULES),
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_CREATE_REWRITE_RULES = 'CREATE_REWRITE_RULES'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_CREATE_REWRITE_RULES: 'Create query rewrite rules',
==============================================================================
CREATE_TABLE
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Aborting Before Step: %s' % step_title(command_steps.STEP_CREATE_TABLE),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                     'error_before_step': step_title(command_steps.STEP_CREATE_TABLE),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Aborting Before Step: %s' % step_title(command_steps.STEP_CREATE_TABLE),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                                '--error-before-step=' + step_title_to_step_id(step_title(command_steps.STEP_CREATE_TABLE))])],
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_CREATE_TABLE = 'CREATE_TABLE'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_CREATE_TABLE: 'Create backend table',
./gluentlib/gluentlib/offload/oracle/oracle_frontend_api.py:CREATE_TABLE_ORACLE_PARTITION_SPEC = "partition_spec"
./gluentlib/gluentlib/offload/oracle/oracle_frontend_api.py:                and CREATE_TABLE_ORACLE_PARTITION_SPEC in table_properties
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_CREATE_TABLE, lambda: self.create_backend_table())
./gluentlib/gluentlib/util/hive_ddl_transform.py:RE_CREATE_TABLE = re.compile('(create\s+)(external\s+)?(table\s+)(if not exists\s+)?(`)?(\w+\.)?(\w+)(`)?(.*)$', re.I | re.S)
./gluentlib/gluentlib/util/hive_ddl_transform.py:        match_ddl = RE_CREATE_TABLE.match(table_ddl)
./gluentlib/gluentlib/util/hive_ddl_transform.py:        elif RE_CREATE_TABLE.match(ddl):
==============================================================================
CREATE_USER_VIEW
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_COMPACT_CREATE_USER_VIEW = 'CREATE_USER_VIEW'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_COMPACT_CREATE_USER_VIEW: 'Create external user view',
==============================================================================
DELTA_COUNT
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_COMPACT_DELTA_COUNT = 'DELTA_COUNT'
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_EXTRACT_DELTA_COUNT1 = 'DELTA_COUNT1'
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_EXTRACT_DELTA_COUNT2 = 'DELTA_COUNT2'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_COMPACT_DELTA_COUNT: 'Check delta table row count before compaction',
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_EXTRACT_DELTA_COUNT1: 'Fetch delta table row count before change extraction',
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_EXTRACT_DELTA_COUNT2: 'Fetch delta table row count after change extraction',
==============================================================================
DELTA_COUNT1
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_EXTRACT_DELTA_COUNT1 = 'DELTA_COUNT1'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_EXTRACT_DELTA_COUNT1: 'Fetch delta table row count before change extraction',
==============================================================================
DELTA_COUNT2
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_EXTRACT_DELTA_COUNT2 = 'DELTA_COUNT2'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_EXTRACT_DELTA_COUNT2: 'Fetch delta table row count after change extraction',
==============================================================================
DELTA_STATS
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_EXTRACT_DELTA_STATS = 'DELTA_STATS'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_EXTRACT_DELTA_STATS: 'Compute delta table stats',
==============================================================================
DEPENDENT_HYBRID_OBJECTS
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_DEPENDENT_HYBRID_OBJECTS = 'DEPENDENT_HYBRID_OBJECTS'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_DEPENDENT_HYBRID_OBJECTS: 'Processing dependent hybrid objects',
==============================================================================
DEPENDENT_HYBRID_VIEWS
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_DEPENDENT_HYBRID_VIEWS = 'DEPENDENT_HYBRID_VIEWS'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_DEPENDENT_HYBRID_VIEWS: 'Generating dependent views in hybrid schema',
./gluentlib/gluentlib/offload/offload_source_table.py:HYBRID_DEPENDENT_HYBRID_VIEWS = ['HYBRID_VIEW', 'JOIN_VIEW', 'AAPD_VIEW']
==============================================================================
DEPENDENT_SCHEMA_OBJECTS
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Skipping Step: %s' % step_title(command_steps.STEP_DEPENDENT_SCHEMA_OBJECTS),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                     'skip': [step_title_to_step_id(step_title(command_steps.STEP_DEPENDENT_SCHEMA_OBJECTS))],
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                                                    step_title(command_steps.STEP_DEPENDENT_SCHEMA_OBJECTS)),
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_DEPENDENT_SCHEMA_OBJECTS = 'DEPENDENT_SCHEMA_OBJECTS'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_DEPENDENT_SCHEMA_OBJECTS: 'Schemas with dependent stored objects',
==============================================================================
DROP_BACKEND_OBJECTS
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_DROP_BACKEND = 'DROP_BACKEND_OBJECTS'
==============================================================================
DROP_CHANGELOG
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_EXTRACT_DROP_RDBMS_LOG = 'DROP_CHANGELOG'
==============================================================================
DROP_EXTERNAL_TABLE
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_DROP_EXTERNAL_TABLE = 'DROP_EXTERNAL_TABLE'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_DROP_EXTERNAL_TABLE: 'Remove existing external table',
==============================================================================
DROP_OBSOLETE_METADATA
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_DROP_OBSOLETE_METADATA = 'DROP_OBSOLETE_METADATA'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_DROP_OBSOLETE_METADATA: 'Delete metadata for obsolete hybrid views',
==============================================================================
DROP_REWRITE_RULES
==============================================================================
DROP_SYNONYM
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_DROP_SYNONYM = 'DROP_SYNONYM'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_DROP_SYNONYM: 'Drop existing hybrid synonym',
==============================================================================
DROP_TABLE
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_DROP_TABLE = 'DROP_TABLE'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_DROP_TABLE: 'Drop backend table',
./gluentlib/gluentlib/offload/backend_table.py:            self._offload_step(command_steps.STEP_DROP_TABLE, lambda: self.drop_table(purge=purge))
==============================================================================
DROP_USER_VIEW
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_COMPACT_DROP_USER_VIEW = 'DROP_USER_VIEW'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_COMPACT_DROP_USER_VIEW: 'Drop user view',
==============================================================================
ESTIMATING_STATS_FULL
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_ESTIMATING_STATS_FULL = 'ESTIMATING_STATS_FULL'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_ESTIMATING_STATS_FULL: 'Generating stats by FULL scan',
==============================================================================
ESTIMATING_STATS_SAMPLE
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_ESTIMATING_STATS = 'ESTIMATING_STATS_SAMPLE'
==============================================================================
EXTRACT_CHANGES
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_EXTRACT_CHANGES = 'EXTRACT_CHANGES'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_EXTRACT_CHANGES: 'Extract changes',
==============================================================================
FETCH_SCN
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_EXTRACT_FETCH_SCN = 'FETCH_SCN'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_EXTRACT_FETCH_SCN: 'Fetch extractor statistics',
==============================================================================
FINAL_LOAD
./gluentlib/tests/integration/persistence/test_orchestration_repo_client.py:            command_steps.STEP_FINAL_LOAD,
./gluentlib/tests/integration/test_sets/stories/test_stories/offload_esoteric.py:    if messages_step_executions(test.offload_messages, step_title(command_steps.STEP_FINAL_LOAD)) != 2:
./gluentlib/tests/integration/test_sets/stories/test_stories/offload_esoteric.py:        log('Execution count of step "%s" != 2' % step_title(command_steps.STEP_FINAL_LOAD))
./gluentlib/tests/integration/test_sets/stories/test_stories/offload_esoteric.py:                         Chunk by size results in 2 chunks, verified by checking STEP_FINAL_LOAD count""",
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Aborting Before Step: %s' % step_title(command_steps.STEP_FINAL_LOAD),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                     'error_before_step': step_title(command_steps.STEP_FINAL_LOAD),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Aborting Before Step: %s' % step_title(command_steps.STEP_FINAL_LOAD),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                                '--error-before-step=' + step_title_to_step_id(step_title(command_steps.STEP_FINAL_LOAD))])],
./gluentlib/tests/integration/test_sets/stories/test_stories/offload_pbo.py:             (lambda test: messages_step_executions(test.offload_messages, step_title(command_steps.STEP_FINAL_LOAD)),
./gluentlib/tests/integration/test_sets/stories/test_stories/offload_pbo.py:             (lambda test: messages_step_executions(test.offload_messages, step_title(command_steps.STEP_FINAL_LOAD)),
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_FINAL_LOAD = 'FINAL_LOAD'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_FINAL_LOAD: 'Load staged data',
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_FINAL_LOAD, lambda: self.load_final_table(sync=sync))
==============================================================================
FINAL_MERGE
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_FINAL_MERGE = 'FINAL_MERGE'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_FINAL_MERGE: 'Merge staged data',
==============================================================================
FIND_JOIN_DATA
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_FIND_JOIN_DATA = 'FIND_JOIN_DATA'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_FIND_JOIN_DATA: 'Find partitions to present',
==============================================================================
FIND_MJOIN_DATA
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_FIND_MJOIN_DATA = 'FIND_MJOIN_DATA'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_FIND_MJOIN_DATA: 'Find partitions to materialize',
==============================================================================
FIND_MODIFIED_PARTITIONS
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_COMPACT_ID_PARTITIONS = 'FIND_MODIFIED_PARTITIONS'
==============================================================================
FIND_OFFLOAD_DATA
./scripts/gluent.py:      command_steps.STEP_FIND_OFFLOAD_DATA,
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_FIND_OFFLOAD_DATA = 'FIND_OFFLOAD_DATA'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_FIND_OFFLOAD_DATA: 'Find data to offload',
==============================================================================
GENERAL_AAPD
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Skipping Step: %s' % step_title(command_steps.STEP_GENERAL_AAPD),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                     'skip': [step_title_to_step_id(step_title(command_steps.STEP_GENERAL_AAPD))],
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:             (lambda test: messages_step_executions(test.offload_messages, step_title(command_steps.STEP_GENERAL_AAPD)),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                                '--skip-steps=' + step_title_to_step_id(step_title(command_steps.STEP_GENERAL_AAPD)) +
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_GENERAL_AAPD = 'GENERAL_AAPD'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_GENERAL_AAPD: 'Create general aggregation rule',
==============================================================================
GET_BACKEND_STATS
./gluentlib/tests/integration/test_sets/stories/test_stories/schema_sync.py:             (lambda test: messages_step_executions(test.offload_messages, step_title(command_steps.STEP_GET_BACKEND_STATS)),
./gluentlib/tests/integration/test_sets/stories/test_stories/schema_sync.py:                                                    step_title(command_steps.STEP_GET_BACKEND_STATS)),
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_GET_BACKEND_STATS = 'GET_BACKEND_STATS'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_GET_BACKEND_STATS: 'Fetching backend stats',
==============================================================================
GRANT_ADMIN
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_GRANT_ADMIN = 'GRANT_ADMIN'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_GRANT_ADMIN: 'Create grant with grant option',
==============================================================================
GRANT_HYBRID_OBJECTS
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_GRANT_HYBRID_OBJECTS = 'GRANT_HYBRID_OBJECTS'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_GRANT_HYBRID_OBJECTS: 'Grant hybrid objects',
==============================================================================
INSERT_PARTITION
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_COMPACT_INSERT_PARTITION = 'INSERT_PARTITION'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_COMPACT_INSERT_PARTITION: 'Populate partition in new base table from user view',
==============================================================================
INSERT_TABLE
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_COMPACT_INSERT_TABLE = 'INSERT_TABLE'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_COMPACT_INSERT_TABLE: 'Populate new base table from user view',
==============================================================================
LOCK_EXTERNAL_STATS
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_EXT_STATS_LOCK = 'LOCK_EXTERNAL_STATS'
==============================================================================
MATERIALIZE_JOIN
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_MATERIALIZE_JOIN = 'MATERIALIZE_JOIN'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_MATERIALIZE_JOIN: 'Materialize staged data',
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_MATERIALIZE_JOIN,
==============================================================================
MESSAGES
./scripts/schema_sync.py:        messages.offload_step(command_steps.STEP_MESSAGES, messages.log_messages, execute=options.execute)
./gluentlib/tests/unit/offload/test_offload_messages.py:                command_steps.STEP_MESSAGES,
./gluentlib/tests/unit/offload/test_offload_messages.py:            messages_title = command_steps.step_title(command_steps.STEP_MESSAGES)
./gluentlib/tests/unit/offload/test_offload_messages.py:                command_steps.STEP_MESSAGES,
./gluentlib/tests/unit/offload/test_offload_messages.py:                    command_steps.STEP_MESSAGES,
./gluentlib/gluentlib/orchestration/orchestration_runner.py:                command_steps.STEP_MESSAGES,
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_MESSAGES = 'MESSAGES'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_MESSAGES: 'Messages',
==============================================================================
OVERWRITE_PARTITION
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_COMPACT_OVERWRITE_PARTITION = 'OVERWRITE_PARTITION'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_COMPACT_OVERWRITE_PARTITION: 'Compact partition in base table from user view',
==============================================================================
PERSIST_IU_CONFIG
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_PERSIST_CONFIG = 'PERSIST_IU_CONFIG'
==============================================================================
PURGE_ARTIFACTS
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_COMPACT_PURGE_ARTIFACTS = 'PURGE_ARTIFACTS'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_COMPACT_PURGE_ARTIFACTS: 'Purge previous artifacts after compaction',
==============================================================================
PURGE_CHANGELOG
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_EXTRACT_PURGE_CHANGELOG = 'PURGE_CHANGELOG'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_EXTRACT_PURGE_CHANGELOG: 'Purge changelog',
==============================================================================
REGISTER_DATA_GOV
./gluentlib/tests/integration/test_sets/stories/story_assertion_functions.py:                                'Error in "%s"' % command_steps.step_title(command_steps.STEP_REGISTER_DATA_GOV))
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_REGISTER_DATA_GOV = 'REGISTER_DATA_GOV'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_REGISTER_DATA_GOV: 'Register data governance metadata',
./gluentlib/gluentlib/data_governance/hadoop_data_governance.py:        return messages.offload_step(command_steps.STEP_REGISTER_DATA_GOV, step_fn,
./gluentlib/gluentlib/data_governance/hadoop_data_governance.py:    return messages.offload_step(command_steps.STEP_REGISTER_DATA_GOV, step_fn,
./gluentlib/gluentlib/data_governance/hadoop_data_governance.py:        return messages.offload_step(command_steps.STEP_REGISTER_DATA_GOV, step_fn,
./gluentlib/gluentlib/data_governance/hadoop_data_governance.py:        return messages.offload_step(command_steps.STEP_REGISTER_DATA_GOV, step_fn,
==============================================================================
RENAME_INITIAL
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_RENAME_INITIAL = 'RENAME_INITIAL'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_RENAME_INITIAL: 'Rename initial base table',
==============================================================================
RENAME_RESET
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_RENAME_RESET = 'RENAME_RESET'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_RENAME_RESET: 'Rename base table',
==============================================================================
RESULT_CACHE_SETUP
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_RESULT_CACHE_SETUP = 'RESULT_CACHE_SETUP'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_RESULT_CACHE_SETUP: 'Setup result cache area',
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_RESULT_CACHE_SETUP, lambda: self.setup_result_cache_area(),
==============================================================================
SAVE_METADATA
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_SAVE_METADATA = 'SAVE_METADATA'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_SAVE_METADATA: 'Save offload metadata',
./gluentlib/gluentlib/offload/offload_metadata_functions.py:    messages.offload_step(command_steps.STEP_SAVE_METADATA,
==============================================================================
SAVE_PARTITIONS
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_COMPACT_SAVE_PARTITIONS = 'SAVE_PARTITIONS'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_COMPACT_SAVE_PARTITIONS: 'Save partition list',
==============================================================================
SET_EXTERNAL_COLUMN_STATS
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_EXT_STATS_SET_COL = 'SET_EXTERNAL_COLUMN_STATS'
==============================================================================
SET_EXTERNAL_STATS
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_EXT_STATS_SET = 'SET_EXTERNAL_STATS'
==============================================================================
SETUP_CHANGELOG
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_SETUP_CHANGELOG = 'SETUP_CHANGELOG'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_SETUP_CHANGELOG: 'Setup changelog',
==============================================================================
SHIP_TO_HDFS
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_EXTRACT_HDFS_SHIP = 'SHIP_TO_HDFS'
==============================================================================
STAGING_CLEANUP
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_STAGING_CLEANUP = 'STAGING_CLEANUP'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_STAGING_CLEANUP: 'Cleanup staging area',
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_STAGING_CLEANUP, lambda: self.cleanup_staging_area())
==============================================================================
STAGING_MINI_CLEANUP
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_STAGING_MINI_CLEANUP = 'STAGING_MINI_CLEANUP'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_STAGING_MINI_CLEANUP: 'Empty staging area',
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_STAGING_MINI_CLEANUP, lambda: self.empty_staging_area(staging_file))
==============================================================================
STAGING_SETUP
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_STAGING_SETUP = 'STAGING_SETUP'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_STAGING_SETUP: 'Setup staging area',
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_STAGING_SETUP, lambda: self.setup_staging_area(staging_file))
==============================================================================
STAGING_TRANSPORT
./gluentlib/tests/integration/persistence/test_orchestration_repo_client.py:            command_steps.STEP_STAGING_TRANSPORT,
./gluentlib/tests/integration/persistence/test_orchestration_repo_client.py:            command_steps.STEP_STAGING_TRANSPORT,
./gluentlib/tests/integration/persistence/test_orchestration_repo_client.py:            command_steps.STEP_STAGING_TRANSPORT,
./gluentlib/tests/integration/test_sets/offload_transport/offload_transport_tests.py:        transport_step = step_title(command_steps.STEP_STAGING_TRANSPORT)
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_STAGING_TRANSPORT = 'STAGING_TRANSPORT'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_STAGING_TRANSPORT: 'Transport data to staging',
./gluentlib/gluentlib/offload/offload_transport.py:        return self._messages.offload_step(command_steps.STEP_STAGING_TRANSPORT, step_fn,
./gluentlib/gluentlib/offload/offload_transport.py:        return self._messages.offload_step(command_steps.STEP_STAGING_TRANSPORT, step_fn,
./gluentlib/gluentlib/offload/offload_transport.py:        return self._messages.offload_step(command_steps.STEP_STAGING_TRANSPORT, step_fn,
./gluentlib/gluentlib/offload/offload_transport.py:        return self._messages.offload_step(command_steps.STEP_STAGING_TRANSPORT, step_fn,
./gluentlib/gluentlib/offload/offload_transport.py:        return self._messages.offload_step(command_steps.STEP_STAGING_TRANSPORT, step_fn,
./gluentlib/gluentlib/offload/offload_transport.py:        rows_imported = self._messages.offload_step(command_steps.STEP_STAGING_TRANSPORT, step_fn,
==============================================================================
UPDATE_USER_VIEW
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_IU_COMPACT_UPDATE_USER_VIEW = 'UPDATE_USER_VIEW'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_IU_COMPACT_UPDATE_USER_VIEW: 'Update external user view',
==============================================================================
VALIDATE_CASTS
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Skipping Step: %s' % step_title(command_steps.STEP_VALIDATE_CASTS),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                     'skip': [step_title_to_step_id(step_title(command_steps.STEP_VALIDATE_CASTS))],
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                                                    step_title(command_steps.STEP_VALIDATE_CASTS)),
./gluentlib/tests/testlib/setup/test_setup_client.py:            offload_options.skip = step_title(command_steps.STEP_VALIDATE_DATA) + ',' + step_title(command_steps.STEP_VALIDATE_CASTS)
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_VALIDATE_CASTS = 'VALIDATE_CASTS'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_VALIDATE_CASTS: 'Validate type conversions',
./gluentlib/gluentlib/offload/backend_table.py:            command_steps.STEP_VALIDATE_CASTS,
==============================================================================
VALIDATE_DATA
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Skipping Step: %s' % step_title(command_steps.STEP_VALIDATE_DATA),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                     'skip': [step_title_to_step_id(step_title(command_steps.STEP_VALIDATE_DATA))],
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                                                    step_title(command_steps.STEP_VALIDATE_DATA)),
./gluentlib/tests/testlib/setup/test_setup_client.py:            offload_options.skip = step_title(command_steps.STEP_VALIDATE_DATA) + ',' + step_title(command_steps.STEP_VALIDATE_CASTS)
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_VALIDATE_DATA = 'VALIDATE_DATA'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_VALIDATE_DATA: 'Validate staged data',
./gluentlib/gluentlib/offload/backend_table.py:            command_steps.STEP_VALIDATE_DATA,
==============================================================================
VERIFY_EXPORTED_DATA
./scripts/gluent.py:    verify_by_count_results = messages.offload_step(command_steps.STEP_VERIFY_EXPORTED_DATA,
./scripts/gluent.py:    if messages.offload_step(command_steps.STEP_VERIFY_EXPORTED_DATA,
./gluentlib/tests/integration/test_sets/stories/test_stories/listener_orchestration.py:        verify_match = get_lines_from_log(command_steps.step_title(command_steps.STEP_VERIFY_EXPORTED_DATA),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Skipping Step: %s' % step_title(command_steps.STEP_VERIFY_EXPORTED_DATA),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                     'skip': [step_title_to_step_id(step_title(command_steps.STEP_VERIFY_EXPORTED_DATA))],
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                                                    step_title(command_steps.STEP_VERIFY_EXPORTED_DATA)),
./gluentlib/gluentlib/orchestration/command_steps.py:STEP_VERIFY_EXPORTED_DATA = 'VERIFY_EXPORTED_DATA'
./gluentlib/gluentlib/orchestration/command_steps.py:    STEP_VERIFY_EXPORTED_DATA: 'Verify exported data',

Same list but excluding entries in the command_steps.py script:

==============================================================================
ALTER_TABLE
./gluentlib/gluentlib/offload/backend_table.py:        return self._offload_step(command_steps.STEP_ALTER_TABLE, step_fn)
==============================================================================
ANALYZE_DATA_TYPES
./scripts/gluent.py:        messages.offload_step(command_steps.STEP_ANALYZE_DATA_TYPES,
==============================================================================
COMPACTION_ADD_DELTA_PARTITION
==============================================================================
COMPACTION_BACKUP_PARTITION
==============================================================================
COMPACTION_BACKUP_TABLE
==============================================================================
COMPUTE_BASE_STATS
==============================================================================
COMPUTE_PART_STATS
==============================================================================
COMPUTE_STATS
./gluentlib/gluentlib/offload/backend_table.py:            self._offload_step(command_steps.STEP_COMPUTE_STATS,
==============================================================================
COPY_EXTERNAL_STATS
==============================================================================
COPY_STATS_TO_BACKEND
./scripts/gluent.py:          command_steps.STEP_COPY_STATS_TO_BACKEND,
./gluentlib/tests/integration/test_sets/stories/test_stories/backend_stats.py:             (lambda test: text_in_warnings(test.offload_messages, step_title(command_steps.STEP_COPY_STATS_TO_BACKEND)),
./gluentlib/tests/integration/test_sets/stories/test_stories/backend_stats.py:             (lambda test: text_in_warnings(test.offload_messages, step_title(command_steps.STEP_COPY_STATS_TO_BACKEND)),
==============================================================================
COUNT_AAPD
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Skipping Step: %s' % step_title(command_steps.STEP_COUNT_AAPD),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                     'skip': [step_title_to_step_id(step_title(command_steps.STEP_COUNT_AAPD))],
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:             (lambda test: messages_step_executions(test.offload_messages, step_title(command_steps.STEP_COUNT_AAPD)),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                                ',' + step_title_to_step_id(step_title(command_steps.STEP_COUNT_AAPD))])],
==============================================================================
CREATE_BASE_TABLE
==============================================================================
CREATE_CONV_VIEW
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_CREATE_CONV_VIEW, step_fn,
==============================================================================
CREATE_DB
./gluentlib/gluentlib/offload/offload_constants.py:""" CAPABILITY_CREATE_DB
./gluentlib/gluentlib/offload/offload_constants.py:CAPABILITY_CREATE_DB = 'create_db'
./gluentlib/gluentlib/offload/offload_constants.py:    CAPABILITY_CREATE_DB: True,
./gluentlib/gluentlib/offload/offload_constants.py:    CAPABILITY_CREATE_DB: True,
./gluentlib/gluentlib/offload/offload_constants.py:    CAPABILITY_CREATE_DB: True,
./gluentlib/gluentlib/offload/offload_constants.py:    CAPABILITY_CREATE_DB: True,
./gluentlib/gluentlib/offload/backend_api.py:    CAPABILITY_CREATE_DB, CAPABILITY_DROP_COLUMN, CAPABILITY_FS_SCHEME_ABFS, CAPABILITY_FS_SCHEME_ADL,\
./gluentlib/gluentlib/offload/backend_api.py:        return self.is_capability_supported(CAPABILITY_CREATE_DB)
./gluentlib/gluentlib/offload/backend_table.py:            self._offload_step(command_steps.STEP_CREATE_DB, lambda: self.create_db())
==============================================================================
CREATE_DELTA_TABLE
==============================================================================
CREATE_EXTERNAL_TABLE
==============================================================================
CREATE_EXTRACTION_TMP_DIR
==============================================================================
CREATE_HDFS_SNAPSHOT
==============================================================================
CREATE_HYBRID_VIEW
./gluentlib/tests/integration/persistence/test_orchestration_repo_client.py:            command_steps.STEP_CREATE_HYBRID_VIEW,
==============================================================================
CREATE_INTERNAL_VIEW
==============================================================================
CREATE_JOIN_VIEW
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_CREATE_JOIN_VIEW, step_fn)
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_CREATE_JOIN_VIEW, step_fn)
==============================================================================
CREATE_REWRITE_RULES
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:             (lambda test: text_in_log(step_title(command_steps.STEP_CREATE_REWRITE_RULES),
==============================================================================
CREATE_TABLE
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Aborting Before Step: %s' % step_title(command_steps.STEP_CREATE_TABLE),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                     'error_before_step': step_title(command_steps.STEP_CREATE_TABLE),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Aborting Before Step: %s' % step_title(command_steps.STEP_CREATE_TABLE),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                                '--error-before-step=' + step_title_to_step_id(step_title(command_steps.STEP_CREATE_TABLE))])],
./gluentlib/gluentlib/offload/oracle/oracle_frontend_api.py:CREATE_TABLE_ORACLE_PARTITION_SPEC = "partition_spec"
./gluentlib/gluentlib/offload/oracle/oracle_frontend_api.py:                and CREATE_TABLE_ORACLE_PARTITION_SPEC in table_properties
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_CREATE_TABLE, lambda: self.create_backend_table())
./gluentlib/gluentlib/util/hive_ddl_transform.py:RE_CREATE_TABLE = re.compile('(create\s+)(external\s+)?(table\s+)(if not exists\s+)?(`)?(\w+\.)?(\w+)(`)?(.*)$', re.I | re.S)
./gluentlib/gluentlib/util/hive_ddl_transform.py:        match_ddl = RE_CREATE_TABLE.match(table_ddl)
./gluentlib/gluentlib/util/hive_ddl_transform.py:        elif RE_CREATE_TABLE.match(ddl):
==============================================================================
CREATE_USER_VIEW
==============================================================================
DELTA_COUNT
==============================================================================
DELTA_COUNT1
==============================================================================
DELTA_COUNT2
==============================================================================
DELTA_STATS
==============================================================================
DEPENDENT_HYBRID_OBJECTS
==============================================================================
DEPENDENT_HYBRID_VIEWS
./gluentlib/gluentlib/offload/offload_source_table.py:HYBRID_DEPENDENT_HYBRID_VIEWS = ['HYBRID_VIEW', 'JOIN_VIEW', 'AAPD_VIEW']
==============================================================================
DEPENDENT_SCHEMA_OBJECTS
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Skipping Step: %s' % step_title(command_steps.STEP_DEPENDENT_SCHEMA_OBJECTS),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                     'skip': [step_title_to_step_id(step_title(command_steps.STEP_DEPENDENT_SCHEMA_OBJECTS))],
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                                                    step_title(command_steps.STEP_DEPENDENT_SCHEMA_OBJECTS)),
==============================================================================
DROP_BACKEND_OBJECTS
==============================================================================
DROP_CHANGELOG
==============================================================================
DROP_EXTERNAL_TABLE
==============================================================================
DROP_OBSOLETE_METADATA
==============================================================================
DROP_REWRITE_RULES
==============================================================================
DROP_SYNONYM
==============================================================================
DROP_TABLE
./gluentlib/gluentlib/offload/backend_table.py:            self._offload_step(command_steps.STEP_DROP_TABLE, lambda: self.drop_table(purge=purge))
==============================================================================
DROP_USER_VIEW
==============================================================================
ESTIMATING_STATS_FULL
==============================================================================
ESTIMATING_STATS_SAMPLE
==============================================================================
EXTRACT_CHANGES
==============================================================================
FETCH_SCN
==============================================================================
FINAL_LOAD
./gluentlib/tests/integration/persistence/test_orchestration_repo_client.py:            command_steps.STEP_FINAL_LOAD,
./gluentlib/tests/integration/test_sets/stories/test_stories/offload_esoteric.py:    if messages_step_executions(test.offload_messages, step_title(command_steps.STEP_FINAL_LOAD)) != 2:
./gluentlib/tests/integration/test_sets/stories/test_stories/offload_esoteric.py:        log('Execution count of step "%s" != 2' % step_title(command_steps.STEP_FINAL_LOAD))
./gluentlib/tests/integration/test_sets/stories/test_stories/offload_esoteric.py:                         Chunk by size results in 2 chunks, verified by checking STEP_FINAL_LOAD count""",
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Aborting Before Step: %s' % step_title(command_steps.STEP_FINAL_LOAD),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                     'error_before_step': step_title(command_steps.STEP_FINAL_LOAD),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Aborting Before Step: %s' % step_title(command_steps.STEP_FINAL_LOAD),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                                '--error-before-step=' + step_title_to_step_id(step_title(command_steps.STEP_FINAL_LOAD))])],
./gluentlib/tests/integration/test_sets/stories/test_stories/offload_pbo.py:             (lambda test: messages_step_executions(test.offload_messages, step_title(command_steps.STEP_FINAL_LOAD)),
./gluentlib/tests/integration/test_sets/stories/test_stories/offload_pbo.py:             (lambda test: messages_step_executions(test.offload_messages, step_title(command_steps.STEP_FINAL_LOAD)),
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_FINAL_LOAD, lambda: self.load_final_table(sync=sync))
==============================================================================
FINAL_MERGE
==============================================================================
FIND_JOIN_DATA
==============================================================================
FIND_MJOIN_DATA
==============================================================================
FIND_MODIFIED_PARTITIONS
==============================================================================
FIND_OFFLOAD_DATA
./scripts/gluent.py:      command_steps.STEP_FIND_OFFLOAD_DATA,
==============================================================================
GENERAL_AAPD
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Skipping Step: %s' % step_title(command_steps.STEP_GENERAL_AAPD),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                     'skip': [step_title_to_step_id(step_title(command_steps.STEP_GENERAL_AAPD))],
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:             (lambda test: messages_step_executions(test.offload_messages, step_title(command_steps.STEP_GENERAL_AAPD)),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                                '--skip-steps=' + step_title_to_step_id(step_title(command_steps.STEP_GENERAL_AAPD)) +
==============================================================================
GET_BACKEND_STATS
./gluentlib/tests/integration/test_sets/stories/test_stories/schema_sync.py:             (lambda test: messages_step_executions(test.offload_messages, step_title(command_steps.STEP_GET_BACKEND_STATS)),
./gluentlib/tests/integration/test_sets/stories/test_stories/schema_sync.py:                                                    step_title(command_steps.STEP_GET_BACKEND_STATS)),
==============================================================================
GRANT_ADMIN
==============================================================================
GRANT_HYBRID_OBJECTS
==============================================================================
INSERT_PARTITION
==============================================================================
INSERT_TABLE
==============================================================================
LOCK_EXTERNAL_STATS
==============================================================================
MATERIALIZE_JOIN
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_MATERIALIZE_JOIN,
==============================================================================
MESSAGES
./scripts/schema_sync.py:        messages.offload_step(command_steps.STEP_MESSAGES, messages.log_messages, execute=options.execute)
./gluentlib/tests/unit/offload/test_offload_messages.py:                command_steps.STEP_MESSAGES,
./gluentlib/tests/unit/offload/test_offload_messages.py:            messages_title = command_steps.step_title(command_steps.STEP_MESSAGES)
./gluentlib/tests/unit/offload/test_offload_messages.py:                command_steps.STEP_MESSAGES,
./gluentlib/tests/unit/offload/test_offload_messages.py:                    command_steps.STEP_MESSAGES,
./gluentlib/gluentlib/orchestration/orchestration_runner.py:                command_steps.STEP_MESSAGES,
==============================================================================
OVERWRITE_PARTITION
==============================================================================
PERSIST_IU_CONFIG
==============================================================================
PURGE_ARTIFACTS
==============================================================================
PURGE_CHANGELOG
==============================================================================
REGISTER_DATA_GOV
./gluentlib/tests/integration/test_sets/stories/story_assertion_functions.py:                                'Error in "%s"' % command_steps.step_title(command_steps.STEP_REGISTER_DATA_GOV))
./gluentlib/gluentlib/data_governance/hadoop_data_governance.py:        return messages.offload_step(command_steps.STEP_REGISTER_DATA_GOV, step_fn,
./gluentlib/gluentlib/data_governance/hadoop_data_governance.py:    return messages.offload_step(command_steps.STEP_REGISTER_DATA_GOV, step_fn,
./gluentlib/gluentlib/data_governance/hadoop_data_governance.py:        return messages.offload_step(command_steps.STEP_REGISTER_DATA_GOV, step_fn,
./gluentlib/gluentlib/data_governance/hadoop_data_governance.py:        return messages.offload_step(command_steps.STEP_REGISTER_DATA_GOV, step_fn,
==============================================================================
RENAME_INITIAL
==============================================================================
RENAME_RESET
==============================================================================
RESULT_CACHE_SETUP
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_RESULT_CACHE_SETUP, lambda: self.setup_result_cache_area(),
==============================================================================
SAVE_METADATA
./gluentlib/gluentlib/offload/offload_metadata_functions.py:    messages.offload_step(command_steps.STEP_SAVE_METADATA,
==============================================================================
SAVE_PARTITIONS
==============================================================================
SET_EXTERNAL_COLUMN_STATS
==============================================================================
SET_EXTERNAL_STATS
==============================================================================
SETUP_CHANGELOG
==============================================================================
SHIP_TO_HDFS
==============================================================================
STAGING_CLEANUP
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_STAGING_CLEANUP, lambda: self.cleanup_staging_area())
==============================================================================
STAGING_MINI_CLEANUP
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_STAGING_MINI_CLEANUP, lambda: self.empty_staging_area(staging_file))
==============================================================================
STAGING_SETUP
./gluentlib/gluentlib/offload/backend_table.py:        self._offload_step(command_steps.STEP_STAGING_SETUP, lambda: self.setup_staging_area(staging_file))
==============================================================================
STAGING_TRANSPORT
./gluentlib/tests/integration/persistence/test_orchestration_repo_client.py:            command_steps.STEP_STAGING_TRANSPORT,
./gluentlib/tests/integration/persistence/test_orchestration_repo_client.py:            command_steps.STEP_STAGING_TRANSPORT,
./gluentlib/tests/integration/persistence/test_orchestration_repo_client.py:            command_steps.STEP_STAGING_TRANSPORT,
./gluentlib/tests/integration/test_sets/offload_transport/offload_transport_tests.py:        transport_step = step_title(command_steps.STEP_STAGING_TRANSPORT)
./gluentlib/gluentlib/offload/offload_transport.py:        return self._messages.offload_step(command_steps.STEP_STAGING_TRANSPORT, step_fn,
./gluentlib/gluentlib/offload/offload_transport.py:        return self._messages.offload_step(command_steps.STEP_STAGING_TRANSPORT, step_fn,
./gluentlib/gluentlib/offload/offload_transport.py:        return self._messages.offload_step(command_steps.STEP_STAGING_TRANSPORT, step_fn,
./gluentlib/gluentlib/offload/offload_transport.py:        return self._messages.offload_step(command_steps.STEP_STAGING_TRANSPORT, step_fn,
./gluentlib/gluentlib/offload/offload_transport.py:        return self._messages.offload_step(command_steps.STEP_STAGING_TRANSPORT, step_fn,
./gluentlib/gluentlib/offload/offload_transport.py:        rows_imported = self._messages.offload_step(command_steps.STEP_STAGING_TRANSPORT, step_fn,
==============================================================================
UPDATE_USER_VIEW
==============================================================================
VALIDATE_CASTS
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Skipping Step: %s' % step_title(command_steps.STEP_VALIDATE_CASTS),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                     'skip': [step_title_to_step_id(step_title(command_steps.STEP_VALIDATE_CASTS))],
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                                                    step_title(command_steps.STEP_VALIDATE_CASTS)),
./gluentlib/tests/testlib/setup/test_setup_client.py:            offload_options.skip = step_title(command_steps.STEP_VALIDATE_DATA) + ',' + step_title(command_steps.STEP_VALIDATE_CASTS)
./gluentlib/gluentlib/offload/backend_table.py:            command_steps.STEP_VALIDATE_CASTS,
==============================================================================
VALIDATE_DATA
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Skipping Step: %s' % step_title(command_steps.STEP_VALIDATE_DATA),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                     'skip': [step_title_to_step_id(step_title(command_steps.STEP_VALIDATE_DATA))],
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                                                    step_title(command_steps.STEP_VALIDATE_DATA)),
./gluentlib/tests/testlib/setup/test_setup_client.py:            offload_options.skip = step_title(command_steps.STEP_VALIDATE_DATA) + ',' + step_title(command_steps.STEP_VALIDATE_CASTS)
./gluentlib/gluentlib/offload/backend_table.py:            command_steps.STEP_VALIDATE_DATA,
==============================================================================
VERIFY_EXPORTED_DATA
./scripts/gluent.py:    verify_by_count_results = messages.offload_step(command_steps.STEP_VERIFY_EXPORTED_DATA,
./scripts/gluent.py:    if messages.offload_step(command_steps.STEP_VERIFY_EXPORTED_DATA,
./gluentlib/tests/integration/test_sets/stories/test_stories/listener_orchestration.py:        verify_match = get_lines_from_log(command_steps.step_title(command_steps.STEP_VERIFY_EXPORTED_DATA),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:         'title': 'Offload Skipping Step: %s' % step_title(command_steps.STEP_VERIFY_EXPORTED_DATA),
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                     'skip': [step_title_to_step_id(step_title(command_steps.STEP_VERIFY_EXPORTED_DATA))],
./gluentlib/tests/integration/test_sets/stories/test_stories/orchestration_step_control.py:                                                    step_title(command_steps.STEP_VERIFY_EXPORTED_DATA)),

@abb9979
Copy link
Contributor

abb9979 commented Nov 23, 2023

The following steps stay:

ALTER_TABLE
ANALYZE_DATA_TYPES
COMPUTE_STATS
COPY_STATS_TO_BACKEND
CREATE_DB
CREATE_TABLE
DROP_TABLE
FINAL_LOAD
FIND_OFFLOAD_DATA
MESSAGES
SAVE_METADATA
STAGING_CLEANUP
STAGING_MINI_CLEANUP
STAGING_SETUP
STAGING_TRANSPORT
VALIDATE_CASTS
VALIDATE_DATA
VERIFY_EXPORTED_DATA

The following steps still have code that needs to be removed (in addition to removal from command_steps.py):

CREATE_CONV_VIEW - backend_table.py
CREATE_JOIN_VIEW - backend_table.py
DEPENDENT_HYBRID_VIEWS - offload_source_table.py
MATERIALIZE_JOIN - backend_table.py
REGISTER_DATA_GOV - hadoop_data_governance.py
RESULT_CACHE_SETUP - backend_table.py

The following steps still have tests that need to be removed (in addition to removal from command_steps.py):

COUNT_AAPD - orchestration_step_control.py
CREATE_HYBRID_VIEW - test_orchestration_repo_client.py
CREATE_REWRITE_RULES - orchestration_step_control.py
DEPENDENT_SCHEMA_OBJECTS - orchestration_step_control.py
GENERAL_AAPD - orchestration_step_control.py
GET_BACKEND_STATS - schema_sync.py
REGISTER_DATA_GOV - story_assertion_functions.py

@nj1973
Copy link
Collaborator Author

nj1973 commented Nov 23, 2023

I would leave the test code alone, it is fine to remove the constants from command_steps.py and leave the tests with broken imports.
Over time I am working through the tests either modernising them or discarding them.
We already have issues to remove Data Governance and Schema Sync code which will also take care of some of these if I don't get there first during my integration test refactoring.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
framework Setup tasks relating to builds, installation and dev initial release Required for initial open source effort
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants