Add new relation source #243

sjanssen2 · 2017-05-09T01:09:50Z

~~Note: this PR will fail until DB is updated for amgut.~~
I have adapted the changes to the DB in this PR for the integration of "source_barcodes_survey"

sjanssen2 · 2017-05-09T01:10:41Z

btw: labadmin and amgut nosetests pass on my local machine

sjanssen2 · 2017-05-09T01:11:12Z

Ready for review @wasade @josenavas

sjanssen2 · 2017-05-09T01:25:35Z

I need some help with one more utf-8 problem:
line 972 in labadmin/knimin/lib/data_access.py: is currently
errors[barcode] = '%s' % e
, which throws the following error on the web-interface. The error can be avoided if I change to
errors[barcode] = '%s' % repr(e)
Is there a nicer way to decode utf-8 here?

AN ERROR HAS OCCURED!

Please copy the following into an email and send this information, along with the url you were trying to access, to... someone in the knight lab
Error

'ascii' codec can't decode byte 0xc5 in position 100: ordinal not in range(128)
Traceback

Traceback (most recent call last):
File "/home/sjanssen/miniconda3/envs/amgut/lib/python2.7/site-packages/tornado/web.py", line 1467, in _execute result = method(*self.path_args, **self.path_kwargs)
File "/media/Daten/AGP/labadmin/knimin/handlers/access_decorators.py", line 55, in post super(DecoratedClass, self).post()
File "/home/sjanssen/miniconda3/envs/amgut/lib/python2.7/site-packages/tornado/web.py", line 2829, in wrapper return method(self, *args, **kwargs)
File "/media/Daten/AGP/labadmin/knimin/handlers/ag_pulldown.py", line 71, in post metadata, failures = db.pulldown(barcodes, blanks, external)
File "/media/Daten/AGP/labadmin/knimin/lib/data_access.py", line 1105, in pulldown external, full)
File "/media/Daten/AGP/labadmin/knimin/lib/data_access.py", line 972, in format_survey_data errors[barcode] = '%s' % e.message.encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 100: ordinal not in range(128)
Request Info

body: barcodes=000067543%2C000067446%2C000067613%2C000067605%2C000067441%2C000067621%2C000067538%2C000067445%2C000067558%2C000067756%2C000067802%2C000067505%2C000067877%2C000067458%2C000067717%2C000067461%2C000067690%2C000067724%2C000067693%2C000067442%2C000067665%2C000067797%2C000067489%2C000067597%2C000067633%2C000067548%2C000067443%2C000067549%2C000067557%2C000067672%2C000067620%2C000067694%2C000067531%2C000067685%2C000067450%2C000067525%2C000067754%2C000067485%2C000067517%2C000067449%2C000067709%2C000067667%2C000067457%2C000067518%2C000067673%2C000067761%2C000067721%2C000067553%2C000067785%2C000067463%2C000067486%2C000067733%2C000067544%2C000067623%2C000067686%2C000067697%2C000067460%2C000067650%2C000067451%2C000067550%2C000067657%2C000067506%2C000067770%2C000067646%2C000067649%2C000067545%2C000067841%2C000067789%2C000067554%2C000067705%2C000067593%2C000067469%2C000067537%2C000067622%2C000067826%2C000067551%2C000067595%2C000067696%2C000067710%2C000067594%2C000067530%2C000067490%2C000067817%2C000067695%2C000067492%2C000067462%2C000067635%2C000067600%2C000067821%2C000067470%2C000067914%2C000067706%2C000067755%2C000067771%2C000068059%2C000067836%2C000067793%2C000067827%2C000067542%2C000067464%2C000067519%2C000067930%2C000067718%2C000067818%2C000067680%2C000067653%2C000067658%2C000067448%2C000067711%2C000067865%2C000067997%2C000067722%2C000067842%2C000067798%2C000067618%2C000067566%2C000067559%2C000067444%2C000067933%2C000067507%2C000067879%2C000067822%2C000068009%2C000067960%2C000067526%2C000067601%2C000067655%2C000067878%2C000067471%2C000067674%2C000067679%2C000067556%2C000067671%2C000067675%2C000067790%2C000067668%2C000067819%2C000067487%2C000067606%2C000067596%2C000067447%2C000067767%2C000067772%2C000067688%2C000068100%2C000067540%2C000067659%2C000067871%2C000067634%2C000068013%2C000067691%2C000067452%2C000067514%2C000068149%2C000067996%2C000067678%2C000067786%2C000067753%2C000067677%2C000067681%2C000067687%2C000067541%2C000067459%2C000067496%2C000067669%2C000067624%2C000067477%2C000067825%2C000067599%2C000067799%2C000068021%2C000067598%2C000067565%2C000067647%2C000067945%2C000067791%2C000067527%2C000067768%2C000067636%2C000067787%2C000067714%2C000067925%2C000067850%2C000067779%2C000067555%2C000067792%2C000067617%2C000067712%2C000067973%2C000068133%2C000067852%2C000067478%2C000067795%2C000068126%2C000067571%2C000067861%2C000067810%2C000067809%2C000067682%2C000067917%2C000068233%2C000068002%2C000068145%2C000067552%2C000067983%2C000067938%2C000067952%2C000067942%2C000067648%2C000067887%2C000067666%2C000067857%2C000067941%2C000067739%2C000067949%2C000067937%2C000067494%2C000067916%2C000067991%2C000067961%2C000067769%2C000067749%2C000068036%2C000067813%2C000067863%2C000067567%2C000067843%2C000067737%2C000068150%2C000067974%2C000067699%2C000068140%2C000067715%2C000067959%2C000067980%2C000068125%2C000067676%2C000068097%2C000067870%2C000067652%2C000067986%2C000067806%2C000069300%2C000067488%2C000067934%2C000067751%2C000067645%2C000067607%2C000068089%2C000067910%2C000067935%2C000068132%2C000067805%2C000067568%2C000067637%2C000067716%2C000067651%2C000068093%2C000067491%2C000068134%2C000067766%2C000068037%2C000068116%2C000067608%2C000068085%2C000068018%2C000067918%2C000067993%2C000067735%2C000067723%2C000068041%2C000067736%2C000067849%2C000068069%2C000067990%2C000067987%2C000067683%2C000067708%2C000068023%2C000068113%2C000067727%2C000067969%2C000067932%2C000067744%2C000067988%2C000067977%2C000067992%2C000068057%2C000068130%2C000067946%2C000068073%2C000067654%2C000067905%2C000067820%2C000068114%2C000068137%2C000067926%2C000067989%2C000068153%2C000067692%2C000067833%2C000067719%2C000069299%2C000067707%2C000067638%2C000067748%2C000067780%2C000067880%2C000067982%2C000067656%2C000068011%2C000068017%2C000067603%2C000067828%2C000067998%2C000067734%2C000067985%2C000067886%2C000067936%2C000067907%2C000067869%2C000067972%2C000067508%2C000067713%2C000067823%2C000067750%2C000067725%2C000067844%2C000067700%2C000067984%2C000068075%2C000067800%2C000067881%2C000068129%2C000067604%2C000068022%2C000068010%2C000068060%2C000068127%2C000067858%2C000068146%2C000068019%2C000067720%2C000068001%2C000067995%2C000067560%2C000068014%2C000067834%2C000068035%2C000067788%2C000067528%2C000067981%2C000067947%2C000067483%2C000068094%2C000067893%2C000068151%2C000067784%2C000068024%2C000067590%2C000067811%2C000067971%2C000069381%2C000067484%2C000067824%2C000067493%2C000067912%2C000067588%2C000067589%2C000067515%2C000067781%2C000067896%2C000067975%2C000067777%2C000067851%2C000068042%2C000067726%2C000067570%2C000067591%2C000067587%2C000068090%2C000068135%2C000068077%2C000067808%2C000068025%2C000067481%2C000069377%2C000068154%2C000067747%2C000067592%2C000067510%2C000067684%2C000067783%2C000067968%2C000067782%2C000068136%2C000067862%2C000068074%2C000067482%2C000067919%2C000067619%2C000067513%2C000067921%2C000067929%2C000067976%2C000068020%2C000067807%2C000067864%2C000067906%2C000067745%2C000068064%2C000067741%2C000067765%2C000067509%2C000067586%2C000067890%2C000067472%2C000067957%2C000067948%2C000067746%2C000067889%2C000067479%2C000067796%2C000068038%2C000067547%2C000067966%2C000067922%2C000067967%2C000067812%2C000068026%2C000069380%2C000067742%2C000067994%2C000067999%2C000067585%2C000068012%2C000067516%2C000067511%2C000067965%2C000067728%2C000067884%2C000067970%2C000068039%2C000067860%2C000068096%2C000067495%2C000067520%2C000068128%2C000068147%2C000068152%2C000068043%2C000068007%2C000067908%2C000068070%2C000068115%2C000067867%2C000067958%2C000067743%2C000068076%2C000068086%2C000068091%2C000067883%2C000067752%2C000068000%2C000067480%2C000068148%2C000067885%2C000067546%2C000067762%2C000068063%2C000067639%2C000067920%2C000068062%2C000067868%2C000067660%2C000068237%2C000067569%2C000068238%2C000068155%2C000067512%2C000068131%2C000067866%2C000067979%2C000067794%2C000068169%2C000068058%2C000067640%2C000068005&blanks=BLANK1.1A%2CBLANK1.1B%2CBLANK1.1C%2CBLANK1.1D%2CBLANK1.1E%2CBLANK1.1F%2CBLANK1.1G%2CBLANK1.1H%2CBLANK2.2A%2CBLANK2.2B%2CBLANK2.2C%2CBLANK2.2D%2CBLANK2.2E%2CBLANK2.2F%2CBLANK2.2G%2CBLANK2.2H%2CBLANK3.3A%2CBLANK3.3B%2CBLANK3.3C%2CBLANK3.3D%2CBLANK3.3E%2CBLANK3.3F%2CBLANK3.3G%2CBLANK3.3H%2CBLANK4.4A%2CBLANK4.4B%2CBLANK4.4C%2CBLANK4.4D%2CBLANK4.4E%2CBLANK4.4F%2CBLANK4.4G%2CBLANK4.4H%2CBLANK5.5A%2CBLANK5.5B%2CBLANK5.5C%2CBLANK5.5D%2CBLANK5.5E%2CBLANK5.5F%2CBLANK5.5G%2CBLANK5.5H%2CBLANK6.6A%2CBLANK6.6B%2CBLANK6.6C%2CBLANK6.6D%2CBLANK6.6E%2CBLANK6.6F%2CBLANK6.6G%2CBLANK6.6H&external=&selected_ag_surveys=-1%2C-2%2C-3%2C-4%2C-5&merged=False
files: {}
protocol: http
connection:
body_arguments: {'barcodes': ['000067543,000067446,000067613,000067605,000067441,000067621,000067538,000067445,000067558,000067756,000067802,000067505,000067877,000067458,000067717,000067461,000067690,000067724,000067693,000067442,000067665,000067797,000067489,000067597,000067633,000067548,000067443,000067549,000067557,000067672,000067620,000067694,000067531,000067685,000067450,000067525,000067754,000067485,000067517,000067449,000067709,000067667,000067457,000067518,000067673,000067761,000067721,000067553,000067785,000067463,000067486,000067733,000067544,000067623,000067686,000067697,000067460,000067650,000067451,000067550,000067657,000067506,000067770,000067646,000067649,000067545,000067841,000067789,000067554,000067705,000067593,000067469,000067537,000067622,000067826,000067551,000067595,000067696,000067710,000067594,000067530,000067490,000067817,000067695,000067492,000067462,000067635,000067600,000067821,000067470,000067914,000067706,000067755,000067771,000068059,000067836,000067793,000067827,000067542,000067464,000067519,000067930,000067718,000067818,000067680,000067653,000067658,000067448,000067711,000067865,000067997,000067722,000067842,000067798,000067618,000067566,000067559,000067444,000067933,000067507,000067879,000067822,000068009,000067960,000067526,000067601,000067655,000067878,000067471,000067674,000067679,000067556,000067671,000067675,000067790,000067668,000067819,000067487,000067606,000067596,000067447,000067767,000067772,000067688,000068100,000067540,000067659,000067871,000067634,000068013,000067691,000067452,000067514,000068149,000067996,000067678,000067786,000067753,000067677,000067681,000067687,000067541,000067459,000067496,000067669,000067624,000067477,000067825,000067599,000067799,000068021,000067598,000067565,000067647,000067945,000067791,000067527,000067768,000067636,000067787,000067714,000067925,000067850,000067779,000067555,000067792,000067617,000067712,000067973,000068133,000067852,000067478,000067795,000068126,000067571,000067861,000067810,000067809,000067682,000067917,000068233,000068002,000068145,000067552,000067983,000067938,000067952,000067942,000067648,000067887,000067666,000067857,000067941,000067739,000067949,000067937,000067494,000067916,000067991,000067961,000067769,000067749,000068036,000067813,000067863,000067567,000067843,000067737,000068150,000067974,000067699,000068140,000067715,000067959,000067980,000068125,000067676,000068097,000067870,000067652,000067986,000067806,000069300,000067488,000067934,000067751,000067645,000067607,000068089,000067910,000067935,000068132,000067805,000067568,000067637,000067716,000067651,000068093,000067491,000068134,000067766,000068037,000068116,000067608,000068085,000068018,000067918,000067993,000067735,000067723,000068041,000067736,000067849,000068069,000067990,000067987,000067683,000067708,000068023,000068113,000067727,000067969,000067932,000067744,000067988,000067977,000067992,000068057,000068130,000067946,000068073,000067654,000067905,000067820,000068114,000068137,000067926,000067989,000068153,000067692,000067833,000067719,000069299,000067707,000067638,000067748,000067780,000067880,000067982,000067656,000068011,000068017,000067603,000067828,000067998,000067734,000067985,000067886,000067936,000067907,000067869,000067972,000067508,000067713,000067823,000067750,000067725,000067844,000067700,000067984,000068075,000067800,000067881,000068129,000067604,000068022,000068010,000068060,000068127,000067858,000068146,000068019,000067720,000068001,000067995,000067560,000068014,000067834,000068035,000067788,000067528,000067981,000067947,000067483,000068094,000067893,000068151,000067784,000068024,000067590,000067811,000067971,000069381,000067484,000067824,000067493,000067912,000067588,000067589,000067515,000067781,000067896,000067975,000067777,000067851,000068042,000067726,000067570,000067591,000067587,000068090,000068135,000068077,000067808,000068025,000067481,000069377,000068154,000067747,000067592,000067510,000067684,000067783,000067968,000067782,000068136,000067862,000068074,000067482,000067919,000067619,000067513,000067921,000067929,000067976,000068020,000067807,000067864,000067906,000067745,000068064,000067741,000067765,000067509,000067586,000067890,000067472,000067957,000067948,000067746,000067889,000067479,000067796,000068038,000067547,000067966,000067922,000067967,000067812,000068026,000069380,000067742,000067994,000067999,000067585,000068012,000067516,000067511,000067965,000067728,000067884,000067970,000068039,000067860,000068096,000067495,000067520,000068128,000068147,000068152,000068043,000068007,000067908,000068070,000068115,000067867,000067958,000067743,000068076,000068086,000068091,000067883,000067752,000068000,000067480,000068148,000067885,000067546,000067762,000068063,000067639,000067920,000068062,000067868,000067660,000068237,000067569,000068238,000068155,000067512,000068131,000067866,000067979,000067794,000068169,000068058,000067640,000068005'], 'blanks': ['BLANK1.1A,BLANK1.1B,BLANK1.1C,BLANK1.1D,BLANK1.1E,BLANK1.1F,BLANK1.1G,BLANK1.1H,BLANK2.2A,BLANK2.2B,BLANK2.2C,BLANK2.2D,BLANK2.2E,BLANK2.2F,BLANK2.2G,BLANK2.2H,BLANK3.3A,BLANK3.3B,BLANK3.3C,BLANK3.3D,BLANK3.3E,BLANK3.3F,BLANK3.3G,BLANK3.3H,BLANK4.4A,BLANK4.4B,BLANK4.4C,BLANK4.4D,BLANK4.4E,BLANK4.4F,BLANK4.4G,BLANK4.4H,BLANK5.5A,BLANK5.5B,BLANK5.5C,BLANK5.5D,BLANK5.5E,BLANK5.5F,BLANK5.5G,BLANK5.5H,BLANK6.6A,BLANK6.6B,BLANK6.6C,BLANK6.6D,BLANK6.6E,BLANK6.6F,BLANK6.6G,BLANK6.6H'], 'selected_ag_surveys': ['-1,-2,-3,-4,-5'], 'merged': ['False'], 'external': ['']}
uri: /ag_pulldown/download/
query_arguments: {}
_start_time: 1494292847.99
headers: Content-Length: 6509 Accept-Language: de-DE,en-US;q=0.7,en;q=0.3 Accept-Encoding: gzip, deflate Connection: keep-alive Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8 User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0 Host: localhost:8889 Referer: http://localhost:8889/ag_pulldown/ Cookie: skid="2|1:0|10:1494039707|4:skid|16:Im50bl9ldGhzcyI=|a2948b12e428c4d1099b67ec11d6182adf789ce4a3f19b2e07e27d9835252c91"; user="2|1:0|10:1494292840|4:user|12:InN0ZWZhbiI=|54daf5123fc1de71afc28adcc0044c6d69487e6cd43e1f42fd3409eac407e4f7" Upgrade-Insecure-Requests: 1 Content-Type: application/x-www-form-urlencoded
host: localhost:8889
version: HTTP/1.1
arguments: {'external': [''], 'blanks': ['BLANK1.1A,BLANK1.1B,BLANK1.1C,BLANK1.1D,BLANK1.1E,BLANK1.1F,BLANK1.1G,BLANK1.1H,BLANK2.2A,BLANK2.2B,BLANK2.2C,BLANK2.2D,BLANK2.2E,BLANK2.2F,BLANK2.2G,BLANK2.2H,BLANK3.3A,BLANK3.3B,BLANK3.3C,BLANK3.3D,BLANK3.3E,BLANK3.3F,BLANK3.3G,BLANK3.3H,BLANK4.4A,BLANK4.4B,BLANK4.4C,BLANK4.4D,BLANK4.4E,BLANK4.4F,BLANK4.4G,BLANK4.4H,BLANK5.5A,BLANK5.5B,BLANK5.5C,BLANK5.5D,BLANK5.5E,BLANK5.5F,BLANK5.5G,BLANK5.5H,BLANK6.6A,BLANK6.6B,BLANK6.6C,BLANK6.6D,BLANK6.6E,BLANK6.6F,BLANK6.6G,BLANK6.6H'], 'selected_ag_surveys': ['-1,-2,-3,-4,-5'], 'merged': ['False'], 'barcodes': ['000067543,000067446,000067613,000067605,000067441,000067621,000067538,000067445,000067558,000067756,000067802,000067505,000067877,000067458,000067717,000067461,000067690,000067724,000067693,000067442,000067665,000067797,000067489,000067597,000067633,000067548,000067443,000067549,000067557,000067672,000067620,000067694,000067531,000067685,000067450,000067525,000067754,000067485,000067517,000067449,000067709,000067667,000067457,000067518,000067673,000067761,000067721,000067553,000067785,000067463,000067486,000067733,000067544,000067623,000067686,000067697,000067460,000067650,000067451,000067550,000067657,000067506,000067770,000067646,000067649,000067545,000067841,000067789,000067554,000067705,000067593,000067469,000067537,000067622,000067826,000067551,000067595,000067696,000067710,000067594,000067530,000067490,000067817,000067695,000067492,000067462,000067635,000067600,000067821,000067470,000067914,000067706,000067755,000067771,000068059,000067836,000067793,000067827,000067542,000067464,000067519,000067930,000067718,000067818,000067680,000067653,000067658,000067448,000067711,000067865,000067997,000067722,000067842,000067798,000067618,000067566,000067559,000067444,000067933,000067507,000067879,000067822,000068009,000067960,000067526,000067601,000067655,000067878,000067471,000067674,000067679,000067556,000067671,000067675,000067790,000067668,000067819,000067487,000067606,000067596,000067447,000067767,000067772,000067688,000068100,000067540,000067659,000067871,000067634,000068013,000067691,000067452,000067514,000068149,000067996,000067678,000067786,000067753,000067677,000067681,000067687,000067541,000067459,000067496,000067669,000067624,000067477,000067825,000067599,000067799,000068021,000067598,000067565,000067647,000067945,000067791,000067527,000067768,000067636,000067787,000067714,000067925,000067850,000067779,000067555,000067792,000067617,000067712,000067973,000068133,000067852,000067478,000067795,000068126,000067571,000067861,000067810,000067809,000067682,000067917,000068233,000068002,000068145,000067552,000067983,000067938,000067952,000067942,000067648,000067887,000067666,000067857,000067941,000067739,000067949,000067937,000067494,000067916,000067991,000067961,000067769,000067749,000068036,000067813,000067863,000067567,000067843,000067737,000068150,000067974,000067699,000068140,000067715,000067959,000067980,000068125,000067676,000068097,000067870,000067652,000067986,000067806,000069300,000067488,000067934,000067751,000067645,000067607,000068089,000067910,000067935,000068132,000067805,000067568,000067637,000067716,000067651,000068093,000067491,000068134,000067766,000068037,000068116,000067608,000068085,000068018,000067918,000067993,000067735,000067723,000068041,000067736,000067849,000068069,000067990,000067987,000067683,000067708,000068023,000068113,000067727,000067969,000067932,000067744,000067988,000067977,000067992,000068057,000068130,000067946,000068073,000067654,000067905,000067820,000068114,000068137,000067926,000067989,000068153,000067692,000067833,000067719,000069299,000067707,000067638,000067748,000067780,000067880,000067982,000067656,000068011,000068017,000067603,000067828,000067998,000067734,000067985,000067886,000067936,000067907,000067869,000067972,000067508,000067713,000067823,000067750,000067725,000067844,000067700,000067984,000068075,000067800,000067881,000068129,000067604,000068022,000068010,000068060,000068127,000067858,000068146,000068019,000067720,000068001,000067995,000067560,000068014,000067834,000068035,000067788,000067528,000067981,000067947,000067483,000068094,000067893,000068151,000067784,000068024,000067590,000067811,000067971,000069381,000067484,000067824,000067493,000067912,000067588,000067589,000067515,000067781,000067896,000067975,000067777,000067851,000068042,000067726,000067570,000067591,000067587,000068090,000068135,000068077,000067808,000068025,000067481,000069377,000068154,000067747,000067592,000067510,000067684,000067783,000067968,000067782,000068136,000067862,000068074,000067482,000067919,000067619,000067513,000067921,000067929,000067976,000068020,000067807,000067864,000067906,000067745,000068064,000067741,000067765,000067509,000067586,000067890,000067472,000067957,000067948,000067746,000067889,000067479,000067796,000068038,000067547,000067966,000067922,000067967,000067812,000068026,000069380,000067742,000067994,000067999,000067585,000068012,000067516,000067511,000067965,000067728,000067884,000067970,000068039,000067860,000068096,000067495,000067520,000068128,000068147,000068152,000068043,000068007,000067908,000068070,000068115,000067867,000067958,000067743,000068076,000068086,000068091,000067883,000067752,000068000,000067480,000068148,000067885,000067546,000067762,000068063,000067639,000067920,000068062,000067868,000067660,000068237,000067569,000068238,000068155,000067512,000068131,000067866,000067979,000067794,000068169,000068058,000067640,000068005']}
_finish_time: None
query:
path: /ag_pulldown/download/
_cookies: Set-Cookie: skid="2|1:0|10:1494039707|4:skid|16:Im50bl9ldGhzcyI=|a2948b12e428c4d1099b67ec11d6182adf789ce4a3f19b2e07e27d9835252c91" Set-Cookie: user="2|1:0|10:1494292840|4:user|12:InN0ZWZhbiI=|54daf5123fc1de71afc28adcc0044c6d69487e6cd43e1f42fd3409eac407e4f7"
method: POST
remote_ip: 127.0.0.1

wasade · 2017-05-09T01:40:55Z

"Is there a nicer way to decode utf-8 here?"

>>> s = 'é'
>>> type(s)
<type 'str'>
>>> s.decode('utf-8')
u'\xe9'

...python27 is really bad with unicode unfortunately. super annoying

wasade · 2017-05-09T01:43:47Z

knimin/lib/data_access.py

+                 LEFT JOIN ag.source_barcodes_surveys USING (barcode)
+                 LEFT JOIN ag.ag_login_surveys USING (survey_id)
+                 LEFT JOIN ag.ag_login
+                 ON (ag.ag_kit.ag_login_id = ag.ag_login.ag_login_id)


USING (ag_login_id) doesn't work here?

no, because there are three tables holding ag_login_id: ag_kit, ag_login_surveys and ag_login

sjanssen2 · 2017-05-09T02:54:03Z

str(e).decode('utf-8') and e.decode('utf-8') results in the same error :-(

josenavas · 2017-05-09T04:26:12Z

@sjanssen2 :

In [7]: e.message.decode('utf-8')
Out[7]: u'\xe9'

You need to access to the message attribute of the error.

wasade · 2017-05-09T04:48:08Z

@sjanssen2, can you try changing the offending line (from the traceback in the thread):

errors[barcode] = '%s' % e.message.encode('utf-8')

to:

errors[barcode] = '%s' % e.message.decode('utf-8')

It may just be a matter of the wrong method off of message

sjanssen2 · 2017-05-09T11:49:39Z

I changed to errors[barcode] = '%s' % e.message.decode('utf-8'), but it still throws the same error.

coveralls · 2017-05-10T19:24:54Z

Coverage increased (+0.2%) to 93.784% when pulling 16e3151 on sjanssen2:add_source into 2e1d684 on biocore:master.

coveralls · 2017-05-10T20:54:34Z

Coverage increased (+0.1%) to 93.747% when pulling 4dc9313 on sjanssen2:add_source into 2e1d684 on biocore:master.

sjanssen2 · 2017-05-10T21:18:33Z

This is now good for reviews @wasade @josenavas

josenavas

Just one comment

josenavas · 2017-05-10T21:58:45Z

knimin/lib/data_access.py

@@ -962,7 +970,10 @@ def _decode_zip_lookup(item):
                        'SURVEY_ID'], unknown_external))
            except Exception as e:
                # Add barcode to error and remove from metadata info
-                errors[barcode] = '%s' % e
+                if isinstance(e, SSLError):


I am surprised about this - SSLError should be a subclass of exception, so it should have the message attribute. Why the special handling?

because it doesn't. I was surprised too.

Weird, it does work on my machine:

In [5]: from requests.exceptions import SSLError In [6]: try: ...: raise SSLError() ...: except Exception as e: ...: pass ...: In [7]: e.message Out[7]: ''

In fact, it works on my laptop and on Travis, but not on flight. I figure it is due to different lib versions. And to be more precise, it seems to be the "message" object itself which does not work as expected if the string is utf-8. Other Exceptions work as expected, but not SSLError.
I'd be happy if we could defer a cleaner solution and continue working on the pulldown.

wasade · 2017-05-10T23:36:14Z

I'm 👍

…in to survey

coveralls · 2017-05-16T17:11:51Z

Coverage increased (+0.2%) to 93.775% when pulling cbd1806 on sjanssen2:add_source into 2e1d684 on biocore:master.

josenavas

Few comments.

josenavas · 2017-05-16T17:56:41Z

knimin/tests/test_ag_add_barcode_kit.py

+                 JOIN ag.ag_kit_barcodes USING (ag_kit_id)
+                 WHERE supplied_kit_id = %s"""
+        barcodes = db._con.execute_fetchall(sql, [supplied_kit_id])
+        if barcodes != []:


Replace by if barcodes

josenavas · 2017-05-16T17:56:51Z

knimin/tests/test_ag_add_barcode_kit.py

+                 WHERE supplied_kit_id = %s AND barcode NOT IN %s"""
+        barcodes = db._con.execute_fetchall(sql, [supplied_kit_id,
+                                                  tuple(old_barcodes)])
+        if barcodes != []:


josenavas · 2017-05-16T17:58:35Z

knimin/tests/test_ag_edit_barcode.py

+                 LEFT JOIN ag.ag_login_surveys USING (ag_login_id)
+                 WHERE barcode = %s"""
+        sourcenames = db._con.execute_fetchall(sql, [barcode])
+        self.assertTrue(sourcenames is not None)


self.assertIsNotNone(sourcenames)

josenavas · 2017-05-16T17:59:43Z

knimin/tests/test_ag_edit_barcode.py

+        response = self.post('/ag_edit_barcode/', payload)
+
+        self.assertEqual(response.code, 200)
+        self.assertEqual(new_sourcename, obs_details['participant_name'])


change to self.assertEqual(obs_details['participant_name'], new_sourcename)
It is expected that the first parameter is the observed value and the second one is the expected one.

josenavas · 2017-05-16T18:01:15Z

knimin/tests/test_ag_edit_barcode.py

@@ -8,6 +8,10 @@


 class TestAGEditBarcodeHandler(TestHandlerBase):
+    # these fields switch None representation and thus are hard to check
+    # for equality
+    none_fields = ['environment_sampled', 'withdrawn', 'refunded']


What do you mean that they change None representation? If they go from some value to None and that's the expected behavior you can check that they're actually None, otherwise they go untested.

I haven't had too much time to investigate this, but I saw that different methods represent the same field differently. Sometimes it's 'N', or None or '' or 'None'.
Since it did not affect my tests here, I did not want to spend more time with it here.

josenavas · 2017-05-16T18:01:58Z

knimin/tests/test_ag_edit_barcode.py

+        # check that no actual change has happened
+
+        dbinfo = db.getAGBarcodeDetails(barcode)
+        for field in set(payload.keys()) - set(self.none_fields):


You can create a copy of payload in which you change this values to None, so you actually test that those values are None as you expect, rather than not testing them.

what do you think about this approach:

if field in self.none_fields: if details[field] in [None, 'N', 'None', '']: details[field] = None if dbinfo[field] in [None, 'N', 'None', '']: dbinfo[field] = None self.assertEqual(details[field], dbinfo[field])```

I'm more concerned about the fact that we don't know which value is going to be returned. That can be fatal for assumptions done in the interface using the function db.getAGBarcodeDetails. Specially I'm surprised that the function db.getAGBarcodeDetails is not consistent in the way that it reports values...

I 100% agree that these are serious concerns, but can we please keep this out of the scope of this PR?

josenavas · 2017-05-16T18:02:36Z

knimin/tests/test_ag_edit_barcode.py

+        response = self.post('/ag_edit_barcode/', payload)
+
+        self.assertEqual(response.code, 200)
+        self.assertEqual(None, obs_details['participant_name'])


self.assertIsNone(obs_details['participant_name'])

josenavas · 2017-05-16T18:02:51Z

knimin/tests/test_ag_edit_barcode.py

+
+        self.assertEqual(response.code, 200)
+        self.assertEqual(None, obs_details['participant_name'])
+        self.assertEqual(obs_surveys, None)


self.assertIsNone(obs_surveys)

josenavas · 2017-05-16T18:03:05Z

knimin/tests/test_ag_edit_barcode.py

+        self.assertEqual(response.code, 200)
+        self.assertEqual(None, obs_details['participant_name'])
+        self.assertEqual(obs_surveys, None)
+        self.assertTrue(db.get_barcode_survey(barcode) is not None)


self.assertIsNotNone(db.get_barcode_survey(barcode))

coveralls · 2017-05-16T19:12:33Z

Coverage increased (+0.2%) to 93.775% when pulling 158f6c5 on sjanssen2:add_source into 2e1d684 on biocore:master.

josenavas · 2017-05-16T20:34:12Z

knimin/tests/test_ag_edit_barcode.py

@@ -8,6 +8,10 @@


 class TestAGEditBarcodeHandler(TestHandlerBase):
+    # these fields switch None representation and thus are hard to check
+    # for equality
+    none_fields = ['environment_sampled', 'withdrawn', 'refunded']


EmbrietteH · 2017-05-17T23:19:11Z

@wasade can you check and merge if ok?

coveralls · 2017-05-18T18:10:45Z

Coverage increased (+0.2%) to 93.775% when pulling cc48745 on sjanssen2:add_source into 2e1d684 on biocore:master.

sjanssen2 · 2017-05-18T18:11:53Z

Justine found:
"We are providing pets_freetext_other field. This is likely not super identifying, although it could be."
That should be solved now.

EmbrietteH · 2017-05-18T18:20:42Z

+1 on pets freetext change. @josenavas @wasade are we ok to merge?

josenavas · 2017-05-18T18:25:29Z

Code looks good to me - however I would hold merging until all functionality has been tested and we make sure that everything is working as expected. Does it make sense?

sjanssen2 added 5 commits May 8, 2017 17:10

modifying DB access to support new table source_barcodes_surveys

de9d8c4

fixing empty joins and geocoder formatting issues

44408d9

clean up after test ran

507c249

clean up after test

f6f8b06

survey_id is no longer needed in this table

f11a047

tmp fix for utf8 issue

ba69fad

wasade reviewed May 9, 2017

View reviewed changes

sjanssen2 added 3 commits May 10, 2017 11:23

fixed test for adding barcodes to existing AG kits

c7191f0

proper dealing with utf8 in error messages

fe2a2a5

flake8 + changes def name

16e3151

fixing utf-8 issues

4dc9313

josenavas reviewed May 10, 2017

View reviewed changes

sjanssen2 added 6 commits May 15, 2017 14:26

test that barcodes not assigned to surveys got found

5932a1b

LEFT JOIN to find unassigned barcodes

b1d531a

adding a new test to check if barcodes can be assigned through labadm…

9e3b73d

…in to survey

fix assignment

f87dd69

removes surveys if participant_name is None

4a2d394

added another test for participant_name is None

8e4fb5a

flake8 stuff

cbd1806

josenavas suggested changes May 16, 2017

View reviewed changes

sjanssen2 added 2 commits May 16, 2017 11:27

addressed Jose's comments

aa4a32e

fixed stupid typo

158f6c5

josenavas approved these changes May 16, 2017

View reviewed changes

sjanssen2 added 2 commits May 18, 2017 10:46

some free text is not removed for pulldown

cd94b9e

adding free text field to list of EBI removal fields

cc48745

josenavas merged commit 49dd881 into biocore:master May 25, 2017

Add new relation source #243

Add new relation source #243

Conversation

sjanssen2 commented May 9, 2017 • edited

sjanssen2 commented May 9, 2017

sjanssen2 commented May 9, 2017

sjanssen2 commented May 9, 2017

wasade commented May 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjanssen2 commented May 9, 2017

josenavas commented May 9, 2017

wasade commented May 9, 2017

sjanssen2 commented May 9, 2017

coveralls commented May 10, 2017

coveralls commented May 10, 2017

sjanssen2 commented May 10, 2017

josenavas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wasade commented May 10, 2017

coveralls commented May 16, 2017

josenavas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented May 16, 2017

Choose a reason for hiding this comment

EmbrietteH commented May 17, 2017

coveralls commented May 18, 2017

sjanssen2 commented May 18, 2017

EmbrietteH commented May 18, 2017

josenavas commented May 18, 2017

sjanssen2 commented May 9, 2017 •

edited