NoETL Log Files
When NoETL runs, messages are written to the log file, keeping it up to date on the state of the process. The following document explains the different sections of the log file with some examples.
- datetime - INFO - projectName started using configuration file - configFilePath
- datetime - INFO - Task: task name, with current task: current task (same as the task name), where next task: next task in sequence if current task succeeds, and task description: describes current task
- datetime - INFO - runTask for task: task name, STARTING STEPS: task.start dictionary
- datetime - INFO - Step: step name, with current step: current step (same as the step name), where next steps: step.success dictionary, and step description: describes current step
- datetime - INFO - runStep for step: step name, THREAD: number of threads for cursor, CURSOR_LIST_INDEX: list of the indices of the cursor, CURSOR: list of cursor dates/integers, ACTION: runShell (function to be called when the step is executed)
- datetime - INFO - Running cursorQueue job: task.taskName.step.stepName.cur.cursor
- datetime - INFO - runShell exitCode: exitCode (0 success), Command: command executed
- datetime - INFO - step: step name, execution time is: execution time for current step
- datetime - INFO - task: task name, execution time is: execution time for current task
2015-08-19 16:48:49.195868 - INFO - Step: insertData , where next steps: {u'0': ['outputFile']} , and step description: Create Hive table
2015-08-19 16:48:49.195909 - INFO - runStep for step: insertData THREAD: 0 ; CURSOR_LIST_INDEX: [0, 1, 2] CURSOR: ['201501'] ACTION: runShell
2015-08-19 16:48:49.196142 - INFO - Running cursorQueue job: task.task1.step.insertData.cur.201501
2015-08-19 16:48:58.615404 - INFO - runShell exitCode: 0 Command: hive -e "USE demo; ALTER TABLE TEST ADD IF NOT EXISTS PARTITION (ds = '201501') LOCATION '/tmp/201501';"
2015-08-19 16:48:58.617748 - INFO - runShell exitCode: 0 Command: echo 201501 > "/tmp/201501.txt"
2015-08-19 16:49:09.641680 - INFO - runShell exitCode: 0 Command: hive -e "USE demo; LOAD DATA LOCAL INPATH '/tmp/201501.txt' OVERWRITE INTO TABLE TEST PARTITION (ds='201501');"
2015-08-19 16:54:27.345075 - INFO - runShell exitCode: 0 Command: hive -e "USE demo; SELECT ds,COUNT(*) FROM TEST WHERE ds = '201501' GROUP BY ds;" >> /tmp/cnt.txt
2015-08-19 16:57:58.074423 - INFO - step: insertData execution time is: 0:09:08.878525
- datetime - INFO - ospid: open source PID, total execution time is: execution time, projectName finished with --- exitCode: exitCode (exit); using configuration file - JSONconfigFile