Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Py3 testing cleanups #6616

Merged
merged 15 commits into from Aug 21, 2018
Merged

Py3 testing cleanups #6616

merged 15 commits into from Aug 21, 2018

Conversation

@nsoranzo
Copy link
Member

nsoranzo commented Aug 21, 2018

Small follow-ups on the amazing work of @mvdbeek in #6561, #6565, #6595 and #6605.

See individual commits for details.

xref. #1715

@@ -575,8 +574,8 @@ def __write_workdir_file(self, config_filename, content, context, is_template=Tr
value = fill_template(content, context=context)
else:
value = content
with open(config_filename, "wb") as f:
f.write(smart_str(value))
with open(config_filename, "w") as f:

This comment has been minimized.

Copy link
@mvdbeek

mvdbeek Aug 21, 2018

Member

This was my initial attempt, but that broke the unicode test tool on docker. I might have changed the encoding in the image in the meantime though, I'll double check and point jenkins at your branch.

This comment has been minimized.

Copy link
@mvdbeek

mvdbeek Aug 21, 2018

Member

yeah, this still fails in the docker image:

FAIL
( unicode_stream ) > Test-3 ... galaxy.tools DEBUG 2018-08-21 07:44:42,132 Validated and populated state for tool request (0.047 ms)
galaxy.tools.actions INFO 2018-08-21 07:44:42,147 Handled output named out_file1 for tool unicode_stream (0.587 ms)
galaxy.tools.actions INFO 2018-08-21 07:44:42,149 Added output datasets to history (1.813 ms)
galaxy.tools.actions INFO 2018-08-21 07:44:42,150 Verified access to datasets for Job[unflushed,tool_id=unicode_stream] (0.026 ms)
galaxy.tools.actions INFO 2018-08-21 07:44:42,151 Setup for job Job[unflushed,tool_id=unicode_stream] complete, ready to flush (1.757 ms)
galaxy.tools.actions INFO 2018-08-21 07:44:42,166 Flushed transaction for job Job[id=3,tool_id=unicode_stream] (15.094 ms)
galaxy.tools.execute DEBUG 2018-08-21 07:44:42,167 Tool [unicode_stream] created job [3] (23.101 ms)
galaxy.tools.execute DEBUG 2018-08-21 07:44:42,171 Executed 1 job(s) for tool unicode_stream request: (38.651 ms)
galaxy.jobs.mapper DEBUG 2018-08-21 07:44:42,516 (3) Mapped job to destination id: local
galaxy.jobs.handler DEBUG 2018-08-21 07:44:42,520 (3) Dispatching to local runner
galaxy.jobs DEBUG 2018-08-21 07:44:42,524 (3) Working directory for job is: /tmp/tmp76q95lht/tmpo0l6a5id/tmpvh6dmpqm/database/job_working_directory_ico6qlt6/000/3
galaxy.jobs DEBUG 2018-08-21 07:44:42,526 (3) Persisting job destination (destination id: local)
galaxy.jobs.runners DEBUG 2018-08-21 07:44:42,534 Job [3] queued (13.951 ms)
galaxy.jobs.handler INFO 2018-08-21 07:44:42,540 (3) Job dispatched
galaxy.jobs.runners ERROR 2018-08-21 07:44:42,582 (3) Failure preparing job
Traceback (most recent call last):
  File "/galaxy/lib/galaxy/jobs/runners/__init__.py", line 187, in prepare_job
    job_wrapper.prepare()
  File "/galaxy/lib/galaxy/jobs/__init__.py", line 874, in prepare
    self.command_line, self.extra_filenames, self.environment_variables = tool_evaluator.build()
  File "/galaxy/lib/galaxy/tools/evaluation.py", line 435, in build
    raise e
  File "/galaxy/lib/galaxy/tools/evaluation.py", line 431, in build
    self.__build_config_files()
  File "/galaxy/lib/galaxy/tools/evaluation.py", line 504, in __build_config_files
    self.__write_workdir_file(config_filename, config_text, param_dict, is_template=is_template)
  File "/galaxy/lib/galaxy/tools/evaluation.py", line 578, in __write_workdir_file
    f.write(value)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-5: ordinal not in range(128)

This comment has been minimized.

Copy link
@nsoranzo

nsoranzo Aug 21, 2018

Author Member

I've added a debug statement and found out that f.encoding is ANSI_X3.4-1968 when run through Docker.

This comment has been minimized.

Copy link
@nsoranzo

nsoranzo Aug 21, 2018

Author Member

Fixed using io.open(config_filename, "w", encoding='utf-8') .

@@ -24,10 +25,10 @@ def nottest(x):


# Simple workflow that takes an input and call cat wrapper on it.
workflow_str = resource_string(__name__, "data/test_workflow_1.ga").decode('utf-8')
workflow_str = unicodify(resource_string(__name__, "data/test_workflow_1.ga"))

This comment has been minimized.

Copy link
@mvdbeek

mvdbeek Aug 21, 2018

Member

I guess you'll want to convert them all, there are more in

lib/galaxy/exceptions/error_codes.py
45:error_codes_json = resource_string(__name__, 'error_codes.json').decode("UTF-8")

lib/galaxy/jobs/runners/util/job_script/__init__.py
16:    resource_string(__name__, 'DEFAULT_JOB_FILE_TEMPLATE.sh').decode('UTF-8')
20:    resource_string(__name__, 'CLUSTER_SLOTS_STATEMENT.sh').decode('UTF-8')
23:    resource_string(__name__, 'MEMORY_STATEMENT.sh').decode('UTF-8')

This comment has been minimized.

Copy link
@nsoranzo

nsoranzo Aug 21, 2018

Author Member

Fixed those too, thanks!

for farm in farms:
farm = unicodify(farms)

This comment has been minimized.

Copy link
@mvdbeek

mvdbeek Aug 21, 2018

Member

This should be farm = unicodify(farm)

This comment has been minimized.

Copy link
@nsoranzo

nsoranzo Aug 21, 2018

Author Member

Fixed, thanks!

@mvdbeek

This comment has been minimized.

Copy link
Member

mvdbeek commented Aug 21, 2018

I've pointed jenkins are your branch, but you still have to trigger manually

@nsoranzo nsoranzo force-pushed the nsoranzo:followup_on_6561 branch from cbd486b to 437eccd Aug 21, 2018
@nsoranzo nsoranzo force-pushed the nsoranzo:followup_on_6561 branch from 437eccd to 05999f9 Aug 21, 2018
@@ -1,443 +0,0 @@
from __future__ import print_function

This comment has been minimized.

Copy link
@jmchilton
This fixes the following traceback:
```
( unicode_stream ) > Test-1 ... galaxy.tools DEBUG 2018-08-21 09:40:59,052 Validated and populated state for tool request (0.098 ms)
galaxy.tools.actions INFO 2018-08-21 09:40:59,076 Handled output named out_file1 for tool unicode_stream (1.337 ms)
galaxy.tools.actions INFO 2018-08-21 09:40:59,080 Added output datasets to history (4.399 ms)
galaxy.tools.actions INFO 2018-08-21 09:40:59,084 Verified access to datasets for Job[unflushed,tool_id=unicode_stream] (0.058 ms)
galaxy.tools.actions INFO 2018-08-21 09:40:59,086 Setup for job Job[unflushed,tool_id=unicode_stream] complete, ready to flush (5.220 ms)
galaxy.tools.actions INFO 2018-08-21 09:40:59,127 Flushed transaction for job Job[id=354,tool_id=unicode_stream] (40.627 ms)
galaxy.tools.execute DEBUG 2018-08-21 09:40:59,127 Tool [unicode_stream] created job [354] (56.506 ms)
galaxy.tools.execute DEBUG 2018-08-21 09:40:59,132 Executed 1 job(s) for tool unicode_stream request: (79.434 ms)
galaxy.jobs.mapper DEBUG 2018-08-21 09:40:59,556 (354) Mapped job to destination id: local
galaxy.jobs.handler DEBUG 2018-08-21 09:40:59,561 (354) Dispatching to local runner
galaxy.jobs DEBUG 2018-08-21 09:40:59,571 (354) Working directory for job is: /tmp/tmp3drfmr6x/tmp_ywd33ar/tmpbzo4a1d9/database/job_working_directory_azry9a1y/000/354
galaxy.jobs DEBUG 2018-08-21 09:40:59,574 (354) Persisting job destination (destination id: local)
galaxy.jobs.runners DEBUG 2018-08-21 09:40:59,589 Job [354] queued (27.361 ms)
galaxy.jobs.handler INFO 2018-08-21 09:40:59,604 (354) Job dispatched
galaxy.jobs.runners ERROR 2018-08-21 09:40:59,715 (354) Failure preparing job
Traceback (most recent call last):
  File "/galaxy/lib/galaxy/jobs/runners/__init__.py", line 187, in prepare_job
    job_wrapper.prepare()
  File "/galaxy/lib/galaxy/jobs/__init__.py", line 874, in prepare
    self.command_line, self.extra_filenames, self.environment_variables = tool_evaluator.build()
  File "/galaxy/lib/galaxy/tools/evaluation.py", line 435, in build
    raise e
  File "/galaxy/lib/galaxy/tools/evaluation.py", line 431, in build
    self.__build_config_files()
  File "/galaxy/lib/galaxy/tools/evaluation.py", line 504, in __build_config_files
    self.__write_workdir_file(config_filename, config_text, param_dict, is_template=is_template)
  File "/galaxy/lib/galaxy/tools/evaluation.py", line 578, in __write_workdir_file
    f.write(value)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-5: ordinal not in range(128)
galaxy.tools.error_reports DEBUG 2018-08-21 09:40:59,869 Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7f92efb81668> generated response None
FAIL
```

which is due to the fact that the default encoding in the Docker image
is `ANSI_X3.4-1968`.

Also update `unicode_stream` test tool to IUC best practices.
@nsoranzo nsoranzo force-pushed the nsoranzo:followup_on_6561 branch from 44d3bf7 to ff0d28c Aug 21, 2018
@mvdbeek mvdbeek merged commit 99517c6 into galaxyproject:dev Aug 21, 2018
5 checks passed
5 checks passed
api test Build finished. 403 tests run, 1 skipped, 0 failed.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
framework test Build finished. 183 tests run, 0 skipped, 0 failed.
Details
integration test Build finished. 114 tests run, 5 skipped, 0 failed.
Details
toolshed test Build finished. 577 tests run, 0 skipped, 0 failed.
Details
@mvdbeek

This comment has been minimized.

Copy link
Member

mvdbeek commented Aug 21, 2018

Awesome, thanks a lot @nsoranzo !

@nsoranzo nsoranzo deleted the nsoranzo:followup_on_6561 branch Aug 21, 2018
@nsoranzo

This comment has been minimized.

Copy link
Member Author

nsoranzo commented Aug 21, 2018

For the record, Py3 Jenkins builds were green too:
api tests on Py3: https://jenkins.galaxyproject.org/job/docker-api-py3/125/
framework tests on Py3: https://jenkins.galaxyproject.org/job/docker-framework-py3/42/
integration tests on Py3: https://jenkins.galaxyproject.org/job/docker-integration-py3/50/

@jmchilton

This comment has been minimized.

Copy link
Member

jmchilton commented Aug 21, 2018

Wonderful, thanks a bunch!

@mvdbeek

This comment has been minimized.

Copy link
Member

mvdbeek commented Aug 21, 2018

Yep, and we now have the triggerphrase galaxybot test py3 if you want to test a PR

@nsoranzo

This comment has been minimized.

Copy link
Member Author

nsoranzo commented Aug 21, 2018

Yep, and we now have the triggerphrase galaxybot test py3 if you want to test a PR

Nice!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.