Skip to content
This repository has been archived by the owner on Jan 13, 2024. It is now read-only.

Commit

Permalink
dernier échec ce soir
Browse files Browse the repository at this point in the history
  • Loading branch information
@sdpython
sdpython committed Nov 25, 2014
1 parent 207aadf commit a049909
Showing 1 changed file with 53 additions and 159 deletions.
212 changes: 53 additions & 159 deletions _doc/notebooks/td3a/td3a_correction_session7az.ipynb
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"metadata": {
"name": "",
"signature": "sha256:031f79a07a32ac4896c0c1677cc99e8fe3a70e0dc5851af8aa383f47d371523d"
"signature": "sha256:6d3f14cc664c66cfc9364a11713ee0b3e5a9ecd98113dfbc7ea0c0ed372d914a"
},
"nbformat": 3,
"nbformat_minor": 0,
Expand Down Expand Up @@ -292,7 +292,7 @@
"source": [
"<h3 id=\"exo1\">Exercice 1 : convertir les valeurs num\u00e9riques</h3>\n",
"\n",
"Le programme suivant prend comme argument les colonnes \u00e0 extraire des fichiers textes qui sont enregistr\u00e9s au format \"python\". Le streaming sur Azure est sensiblement diff\u00e9rent du streaming sur Cloudera. Les scripts python sont interpr\u00e9t\u00e9s avec la machine virtuelle java tout comme pig. La solution suivante s'inspire de [Utilisation de Python avec Hive et Pig dans HDInsight](http://azure.microsoft.com/fr-fr/documentation/articles/hdinsight-python/). Voir \u00e9galement [Writing Jython UDFs](https://help.mortardata.com/technologies/pig/writing_jython_udfs)."
"Le programme suivant prend comme argument les colonnes \u00e0 extraire des fichiers textes qui sont enregistr\u00e9s au format \"python\". Le streaming sur Azure est sensiblement diff\u00e9rent du streaming sur Cloudera. Les scripts python sont interpr\u00e9t\u00e9s avec la machine virtuelle java tout comme pig. La solution suivante s'inspire de [Utilisation de Python avec Hive et Pig dans HDInsight](http://azure.microsoft.com/fr-fr/documentation/articles/hdinsight-python/). Voir \u00e9galement [Writing Jython UDFs](https://help.mortardata.com/technologies/pig/writing_jython_udfs)"
]
},
{
Expand All @@ -308,13 +308,13 @@
" res = [ ]\n",
" for station in js:\n",
" vals = [ str(station[c]) for c in cols ]\n",
" res.append( tuple(vals) \n",
" res.append( tuple(vals) )\n",
" return res"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 91
"prompt_number": 99
},
{
"cell_type": "markdown",
Expand All @@ -340,7 +340,7 @@
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 92
"prompt_number": 100
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -410,19 +410,19 @@
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 95,
"prompt_number": 101,
"text": [
"{'id': 'job_1416874839254_0028'}"
"{'id': 'job_1416874839254_0030'}"
]
}
],
"prompt_number": 95
"prompt_number": 101
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"st = %hd_job_status job_1416874839254_0026\n",
"st = %hd_job_status job_1416874839254_0030\n",
"st[\"id\"],st[\"percentComplete\"],st[\"status\"][\"jobComplete\"]"
],
"language": "python",
Expand All @@ -431,13 +431,13 @@
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 87,
"prompt_number": 103,
"text": [
"('job_1416874839254_0026', '100% complete', False)"
"('job_1416874839254_0030', '0% complete', False)"
]
}
],
"prompt_number": 87
"prompt_number": 103
},
{
"cell_type": "code",
Expand All @@ -451,167 +451,61 @@
{
"html": [
"<pre>\n",
"\tat org.apache.pig.scripting.jython.JythonFunction.exec(JythonFunction.java:117)\n",
"\t... 16 more\n",
"Caused by: java.io.IOException: Mark invalid\n",
"\tat java.io.BufferedReader.reset(BufferedReader.java:505)\n",
"\tat org.python.core.ParserFacade.prepBufReader(ParserFacade.java:276)\n",
"\tat org.python.core.ParserFacade.prepBufReader(ParserFacade.java:349)\n",
"\tat org.python.core.ParserFacade.parse(ParserFacade.java:196)\n",
"\t... 32 more\n",
"\n",
"2014-11-25 02:08:16,020 [main] <b><font color=\"#DD0000\">ERROR</font></b> org.apache.pig.tools.grunt.GruntParser - org.apache.pig.backend.executionengine.ExecException: <b><font color=\"#DD0000\">ERROR</font></b> 2997: Unable to recreate exception from backed error: Error: org.apache.pig.backend.executionengine.ExecException: <b><font color=\"#DD0000\">ERROR</font></b> 0: Error executing function\n",
"\tat org.apache.pig.scripting.jython.JythonFunction.exec(JythonFunction.java:120)\n",
"\tat org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:330)\n",
"\tat org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNextDataBag(POUserFunc.java:374)\n",
"\tat org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:309)\n",
"\tat org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:378)\n",
"\tat org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:298)\n",
"\tat org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:282)\n",
"\tat org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:277)\n",
"\tat org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)\n",
"\tat org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)\n",
"\tat org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)\n",
"\tat org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)\n",
"\tat org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)\n",
"\tat java.security.AccessController.doPrivileged(Native Method)\n",
"\tat javax.security.auth.Subject.doAs(Subject.java:415)\n",
"\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)\n",
"\tat org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)\n",
"Caused by: Traceback (most recent call last):\n",
" File \"D:\\Users\\hdp\\AppData\\Local\\Temp\\pig1248585840543764175tmp\\stream_json.py\", line 5, in extract_columns_from_js\n",
"java.io.IOException: Mark invalid\n",
"\n",
"\tat java.io.BufferedReader.reset(BufferedReader.java:505)\n",
"\n",
"\tat org.python.core.ParserFacade.prepBufReader(ParserFacade.java:276)\n",
"\n",
"\tat org.python.core.ParserFacade.prepBufReader(ParserFacade.java:349)\n",
"\n",
"\tat org.python.core.ParserFacade.parse(ParserFacade.java:196)\n",
"\n",
"\tat org.python.core.CompileFunction.compile(__builtin__.java:1512)\n",
"\n",
"\tat org.python.core.CompileFunction.compile(__builtin__.java:1488)\n",
"\n",
"\tat org.python.core.CompileFunction.compile(__builtin__.java:1480)\n",
"\n",
"\tat org.python.core.__builtin__.eval(__builtin__.java:479)\n",
"\n",
"\tat org.python.core.__builtin__.eval(__builtin__.java:495)\n",
"\n",
"\tat org.python.core.BuiltinFunctions.__call__(__builtin__.java:87)\n",
"\n",
"\tat org.python.core.PyObject.__call__(PyObject.java:391)\n",
"\n",
"\tat org.python.pycode._pyx3.extract_columns_from_js$1(D:\\Users\\hdp\\AppData\\Local\\Temp\\pig1248585840543764175tmp\\stream_json.py:10)\n",
"\n",
"\tat org.python.pycode._pyx3.call_function(D:\\Users\\hdp\\AppData\\Local\\Temp\\pig1248585840543764175tmp\\stream_json.py)\n",
"\n",
"\tat org.python.core.PyTableCode.call(PyTableCode.java:165)\n",
"\n",
"\tat org.python.core.PyBaseCode.call(PyBaseCode.java:301)\n",
"\n",
"\tat org.python.core.PyFunction.function___call__(PyFunction.java:376)\n",
"\n",
"\tat org.python.core.PyFunction.__call__(PyFunction.java:371)\n",
"\n",
"\tat org.python.core.PyFunction.__call__(PyFunction.java:361)\n",
"\n",
"\tat org.python.core.PyFunction.__call__(PyFunction.java:356)\n",
"\n",
"\tat org.apache.pig.scripting.jython.JythonFunction.exec(JythonFunction.java:117)\n",
"\n",
"\tat org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:330)\n",
"\n",
"\tat org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNextDataBag(POUserFunc.java:374)\n",
"\n",
"\tat org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:309)\n",
"\n",
"\tat org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:378)\n",
"\n",
"\tat org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:298)\n",
"\n",
"\tat org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:282)\n",
"\n",
"\tat org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:277)\n",
"\n",
"\tat org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)\n",
"\n",
"\tat org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)\n",
"\n",
"\tat org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)\n",
"\n",
"\tat org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)\n",
"\n",
"\tat org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)\n",
"\n",
"\tat java.security.AccessController.doPrivileged(Native Method)\n",
"\n",
"\tat javax.security.auth.Subject.doAs(Subject.java:415)\n",
"\n",
"\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)\n",
"\n",
"\tat org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)\n",
"\n",
"\n",
"java.io.IOException: java.io.IOException: Mark invalid\n",
"\n",
"\tat org.python.core.Py.JavaError(Py.java:495)\n",
"\tat org.python.core.ParserFacade.fixParseError(ParserFacade.java:104)\n",
"\tat org.python.core.ParserFacade.parse(ParserFacade.java:199)\n",
"\tat org.python.core.CompileFunction.compile(__builtin__.java:1512)\n",
"\tat org.python.core.CompileFunction.compile(__builtin__.java:1488)\n",
"\tat org.python.core.CompileFunction.compile(__builtin__.java:1480)\n",
"\tat org.python.core.__builtin__.eval(__builtin__.java:479)\n",
"\tat org.python.core.__builtin__.eval(__builtin__.java:495)\n",
"\tat org.python.core.BuiltinFunctions.__call__(__builtin__.java:87)\n",
"\tat org.python.core.PyObject.__call__(PyObject.java:391)\n",
"\tat org.python.pycode._pyx3.extract_columns_from_js$1(D:\\Users\\hdp\\AppData\\Local\\Temp\\pig1248585840543764175tmp\\stream_json.py:10)\n",
"\tat org.python.pycode._pyx3.call_function(D:\\Users\\hdp\\AppData\\Local\\Temp\\pig1248585840543764175tmp\\stream_json.py)\n",
"\tat org.python.core.PyTableCode.call(PyTableCode.java:165)\n",
"\tat org.python.core.PyBaseCode.call(PyBaseCode.java:301)\n",
"\tat org.python.core.PyFunction.function___call__(PyFunction.java:376)\n",
"\tat org.python.core.PyFunction.__call__(PyFunction.java:371)\n",
"\tat org.python.core.PyFunction.__call__(PyFunction.java:361)\n",
"\tat org.python.core.PyFunction.__call__(PyFunction.java:356)\n",
"\tat org.apache.pig.scripting.jython.JythonFunction.exec(JythonFunction.java:117)\n",
"\t... 16 more\n",
"Caused by: java.io.IOException: Mark invalid\n",
"\tat java.io.BufferedReader.reset(BufferedReader.java:505)\n",
"\tat org.python.core.ParserFacade.prepBufReader(ParserFacade.java:276)\n",
"\tat org.python.core.ParserFacade.prepBufReader(ParserFacade.java:349)\n",
"\tat org.python.core.ParserFacade.parse(ParserFacade.java:196)\n",
"\t... 32 more\n",
"\n",
"\tat org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:217)\n",
"\tat org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:151)\n",
"\tat org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:429)\n",
"\tat org.apache.pig.PigServer.launchPlan(PigServer.java:1324)\n",
"\tat org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1309)\n",
"\tat org.apache.pig.PigServer.execute(PigServer.java:1299)\n",
"\tat org.apache.pig.PigServer.executeBatch(PigServer.java:377)\n",
"\tat org.apache.pig.PigServer.executeBatch(PigServer.java:355)\n",
"\tat org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)\n",
"\tat org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)\n",
"2014-11-25 02:13:37,539 [main] INFO org.apache.pig.Main - Apache Pig version 0.12.1.2.1.6.0-2103 (r: unknown) compiled Oct 08 2014, 00:25:46\n",
"2014-11-25 02:13:37,539 [main] INFO org.apache.pig.Main - Logging error messages to: C:\\apps\\dist\\hadoop-2.4.0.2.1.6.0-2103\\logs\\pig_1416881617539.log\n",
"2014-11-25 02:13:38,805 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file D:\\Users\\hdp/.pigbootup not found\n",
"2014-11-25 02:13:39,180 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address\n",
"2014-11-25 02:13:39,180 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS\n",
"2014-11-25 02:13:39,180 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: wasb://clusterensaeazure1@hdblobstorage.blob.core.windows.net\n",
"2014-11-25 02:13:39,961 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS\n",
"2014-11-25 02:13:40,414 [main] INFO org.apache.pig.scripting.jython.JythonScriptEngine - created tmp python.cachedir=D:\\Users\\hdp\\AppData\\Local\\Temp\\pig_jython_2963841330580198622\n",
"2014-11-25 02:13:43,305 [main] WARN org.apache.pig.scripting.jython.JythonScriptEngine - pig.cmd.args.remainders is empty. This is not expected unless on testing.\n",
"2014-11-25 02:13:43,492 [main] <b><font color=\"#DD0000\">ERROR</font></b> org.apache.pig.tools.grunt.Grunt - <b><font color=\"#DD0000\">ERROR</font></b> 1121: Python Error. File \"D:\\Users\\hdp\\AppData\\Local\\Temp\\pig7358684366656591264tmp\\stream_json.py\", line 10\n",
" return res\n",
" ^\n",
"SyntaxError: no viable alternative at input 'return'\n",
"\n",
"2014-11-25 02:13:43,492 [main] <b><font color=\"#DD0000\">ERROR</font></b> org.apache.pig.tools.grunt.Grunt - org.apache.pig.backend.executionengine.ExecException: <b><font color=\"#DD0000\">ERROR</font></b> 1121: Python Error. File \"D:\\Users\\hdp\\AppData\\Local\\Temp\\pig7358684366656591264tmp\\stream_json.py\", line 10\n",
" return res\n",
" ^\n",
"SyntaxError: no viable alternative at input 'return'\n",
"\n",
"\tat org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:249)\n",
"\tat org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.init(JythonScriptEngine.java:163)\n",
"\tat org.apache.pig.scripting.jython.JythonScriptEngine.registerFunctions(JythonScriptEngine.java:344)\n",
"\tat org.apache.pig.PigServer.registerCode(PigServer.java:562)\n",
"\tat org.apache.pig.tools.grunt.GruntParser.processRegister(GruntParser.java:476)\n",
"\tat org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:546)\n",
"\tat org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)\n",
"\tat org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)\n",
"\tat org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)\n",
"\tat org.apache.pig.Main.run(Main.java:478)\n",
"\tat org.apache.pig.Main.main(Main.java:156)\n",
"Caused by: File \"D:\\Users\\hdp\\AppData\\Local\\Temp\\pig7358684366656591264tmp\\stream_json.py\", line 10\n",
" return res\n",
" ^\n",
"SyntaxError: no viable alternative at input 'return'\n",
"\n",
"\tat org.python.core.ParserFacade.fixParseError(ParserFacade.java:92)\n",
"\tat org.python.core.ParserFacade.parse(ParserFacade.java:184)\n",
"\tat org.python.core.Py.compile_flags(Py.java:1731)\n",
"\tat org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:235)\n",
"\tat org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:217)\n",
"\t... 10 more\n",
"\n",
"Details also at logfile: C:\\apps\\dist\\hadoop-2.4.0.2.1.6.0-2103\\logs\\pig_1416881208845.log\n",
"Details also at logfile: C:\\apps\\dist\\hadoop-2.4.0.2.1.6.0-2103\\logs\\pig_1416881617539.log\n",
"\n",
"</pre>"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 89,
"prompt_number": 98,
"text": [
"<IPython.core.display.HTML at 0x68b2cb0>"
"<IPython.core.display.HTML at 0x953a910>"
]
}
],
"prompt_number": 89
"prompt_number": 98
},
{
"cell_type": "markdown",
Expand Down

0 comments on commit a049909

Please sign in to comment.