Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Edit New issue how to using hiveQL with params #169

Open
getGitxxx opened this issue Jan 21, 2015 · 1 comment
Open

Edit New issue how to using hiveQL with params #169

getGitxxx opened this issue Jan 21, 2015 · 1 comment

Comments

@getGitxxx
Copy link

hi all
I use hive plugin on azkaban v2.5,I need some dynamic params to hiveQL files
without azkaban,I set hiveconf like this
bin/hive -f “path/to/hiveql/test.q” -hiveconf tabname=table1
in test.q,i use params like this
select * from ${hiveconf:tabname}
this hive job works well.

so .my requestion is how to use hive like this in Azkaban?

Any relay will greatly appreciated!
@getGitxxx
Copy link
Author

this is log info

22-01-2015 18:14:21 CST hive-demo INFO - Starting job hive-demo at 1421921661351
22-01-2015 18:14:21 CST hive-demo INFO - Building hive job executor.
22-01-2015 18:14:21 CST hive-demo INFO - Not setting up secure proxy info for child process
22-01-2015 18:14:21 CST hive-demo INFO - No classpath specified. Trying to load classes from /app/azkaban/azkaban-executor-2.5.0/executions/149
22-01-2015 18:14:21 CST hive-demo INFO - 1 commands to execute.
22-01-2015 18:14:21 CST hive-demo INFO - Command: java -Dhive.querylog.location=. -Dhive.exec.scratchdir=/tmp/hive-azkaban -Dhive.aux.jars.path=file:///home/sinova/hive-0.14.0/aux/lib -Xms64M -Xmx256M -cp /app/azkaban/azkaban-executor-2.5.0/lib/azkaban-2.5.0.jar:/app/azkaban/azkaban-executor-2.5.0/plugins/jobtypes/hive/azkaban-jobtype-2.5.jar:/app/azkaban/azkaban-executor-2.5.0/plugins/jobtypes/hive/azkaban-hadoopsecuritymanager-2.5.jar:/home/sinova/hadoop-1.0.1/conf:/home/sinova/hadoop-1.0.1/lib/:/home/sinova/hive-0.14.0/lib/:/home/sinova/hive-0.14.0/conf:/app/azkaban/azkaban-executor-2.5.0/plugins/jobtypes/hive/file:/home/sinova/hive-0.14.0/aux/lib:/home/sinova/hadoop-1.0.1/hadoop-core-1.0.1.jar azkaban.jobtype.HadoopSecureHiveWrapper -hiveconf 'dt=20150122' -f hive-load-part.hql
22-01-2015 18:14:21 CST hive-demo INFO - Environment variables: {JOB_NAME=hive-demo, HIVE_HOME=$HIVE_HOME, JOB_PROP_FILE=/app/azkaban/azkaban-executor-2.5.0/executions/149/hive-demo_props_1032021434949287271_tmp, JOB_OUTPUT_PROP_FILE=/app/azkaban/azkaban-executor-2.5.0/executions/149/hive-demo_output_4340661934105694333_tmp, HIVE_AUX_JARS_PATH=file:///home/sinova/hive-0.14.0/aux/lib}
22-01-2015 18:14:21 CST hive-demo INFO - Working directory: /app/azkaban/azkaban-executor-2.5.0/executions/149
22-01-2015 18:14:21 CST hive-demo INFO - 2015/01/22 18:14:21.899 +0800 INFO [root] [Azkaban] Not proxying.
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.280 +0800 INFO [root] [Azkaban] HiveConf = Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@249ef1e, file:/home/sinova/hive-0.14.0/conf/hive-site.xml
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.280 +0800 INFO [root] [Azkaban] According to the conf, we're talking to the Hive hosted at: jdbc:mysql://10.20.11.17:3306/hive13?createDatabaseIfNotExist=true
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.280 +0800 INFO [root] [Azkaban] No files in to expand in aux jar path. Returning original parameter
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.280 +0800 INFO [root] [Azkaban] Hive aux jars variable not expanded
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.289 +0800 INFO [root] [Azkaban] Got auxJars = file:///home/sinova/hive-0.14.0/aux/lib
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.454 +0800 INFO [metastore] [Azkaban] Trying to connect to metastore with URI thrift://10.40.33.11:9083
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.504 +0800 INFO [metastore] [Azkaban] Connected to metastore.
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.824 +0800 WARN [NativeCodeLoader] [Azkaban] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.828 +0800 INFO [SessionState] [Azkaban] Created local directory: /tmp/38c7abd4-70c2-41ba-94c5-541b95722644_resources
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.881 +0800 INFO [SessionState] [Azkaban] Created HDFS directory: /tmp/hive-azkaban/sinova/38c7abd4-70c2-41ba-94c5-541b95722644
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.885 +0800 INFO [SessionState] [Azkaban] Created local directory: /tmp/sinova/38c7abd4-70c2-41ba-94c5-541b95722644
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.893 +0800 INFO [SessionState] [Azkaban] Created HDFS directory: /tmp/hive-azkaban/sinova/38c7abd4-70c2-41ba-94c5-541b95722644/_tmp_space.db
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.895 +0800 INFO [SessionState] [Azkaban] No Tez session required at this point. hive.execution.engine=mr.
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.896 +0800 INFO [root] [Azkaban] SessionState = org.apache.hadoop.hive.cli.CliSessionState@4777eb45
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.897 +0800 INFO [root] [Azkaban] Executing query: hive-load-part.hql
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.931 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.931 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.932 +0800 INFO [Driver] [Azkaban] Concurrency mode is disabled, not creating a lock manager
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.932 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.975 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:22 CST hive-demo INFO - 2015/01/22 18:14:22.984 +0800 INFO [ParseDriver] [Azkaban] Parsing command: LOAD DATA LOCAL INPATH '/app/azkaban/web2' OVERWRITE INTO TABLE words PARTITION (day_id='${hiveconf:dt}')
22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.376 +0800 INFO [ParseDriver] [Azkaban] Parse Completed
22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.377 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.381 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.631 +0800 INFO [Driver] [Azkaban] Semantic Analysis Completed
22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.631 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.639 +0800 INFO [Driver] [Azkaban] Returning Hive schema: Schema(fieldSchemas:null, properties:null)
22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.639 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.639 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.639 +0800 INFO [Driver] [Azkaban] Starting command: LOAD DATA LOCAL INPATH '/app/azkaban/web2' OVERWRITE INTO TABLE words PARTITION (day_id='${hiveconf:dt}')
22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.644 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.644 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.644 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.645 +0800 INFO [Driver] [Azkaban] Starting task [Stage-0:MOVE] in serial mode
22-01-2015 18:14:23 CST hive-demo ERROR - Loading data to table default.words partition (day_id=${hiveconf:dt})
22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.646 +0800 INFO [Task] [Azkaban] Loading data to table default.words partition (day_id=${hiveconf:dt}) from file:/app/azkaban/web2
22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.687 +0800 INFO [MoveTask] [Azkaban] Partition is: {day_id=${hiveconf:dt}}
22-01-2015 18:14:23 CST hive-demo INFO - 2015/01/22 18:14:23.954 +0800 INFO [Hive] [Azkaban] Replacing src:file:/app/azkaban/web2;dest: hdfs://bj-ecs-jm429-I-e2-1-bl460c-9-null-9:9000/user/sinova/hive/warehouse/words/day_id=$%7Bhiveconf%3Adt}/web2;Status:true
22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.255 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.256 +0800 INFO [Driver] [Azkaban] Starting task [Stage-1:STATS] in serial mode
22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.256 +0800 INFO [StatsTask] [Azkaban] Executing stats task
22-01-2015 18:14:24 CST hive-demo ERROR - Partition default.words{day_id=${hiveconf:dt}} stats: [numFiles=1, numRows=0, totalSize=2728995, rawDataSize=0]
22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.368 +0800 INFO [Task] [Azkaban] Partition default.words{day_id=${hiveconf:dt}} stats: [numFiles=1, numRows=0, totalSize=2728995, rawDataSize=0]
22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.508 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.508 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:24 CST hive-demo ERROR - OK
22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.509 +0800 INFO [Driver] [Azkaban] OK
22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.510 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.510 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.510 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:24 CST hive-demo ERROR - Time taken: 1.583 seconds
22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.512 +0800 INFO [CliDriver] [Azkaban] Time taken: 1.583 seconds
22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.512 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:24 CST hive-demo INFO - 2015/01/22 18:14:24.512 +0800 INFO [PerfLogger] [Azkaban]
22-01-2015 18:14:24 CST hive-demo INFO - Process completed successfully in 3 seconds.
22-01-2015 18:14:24 CST hive-demo INFO - Finishing job hive-demo at 1421921664644 with status SUCCEEDED

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@getGitxxx