New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes to logic in setup management command for loading content packs. #5065

Merged
merged 2 commits into from Apr 1, 2016

Conversation

Projects
None yet
2 participants
@jamalex
Member

jamalex commented Mar 31, 2016

Summary

@aronasorman knows what's what, and I gotta run. Can add more notes later as needed.

@jamalex jamalex added this to the 0.16.1 milestone Mar 31, 2016

@jamalex jamalex added the has PR label Mar 31, 2016

@aronasorman

This comment has been minimized.

Show comment
Hide comment
@aronasorman

aronasorman Mar 31, 2016

Member

This is the situation as I understand it. @jamalex feel free to correct me:

Currently, both the PyPI and deb installer will always ask if a user wishes to download and install en.zip, even if khan_assessment.zip are installed by the installer/installed beforehand.

They best solution we arrived to, is to reinstate the assessmentitems.version file in both en.zip and khan_assessment.zip, and then have the setup process check for the existence of said file and the version contained, to see if it's up-to-date. If not, then we ask if they want to update their content pack.

Member

aronasorman commented Mar 31, 2016

This is the situation as I understand it. @jamalex feel free to correct me:

Currently, both the PyPI and deb installer will always ask if a user wishes to download and install en.zip, even if khan_assessment.zip are installed by the installer/installed beforehand.

They best solution we arrived to, is to reinstate the assessmentitems.version file in both en.zip and khan_assessment.zip, and then have the setup process check for the existence of said file and the version contained, to see if it's up-to-date. If not, then we ask if they want to update their content pack.

@aronasorman

This comment has been minimized.

Show comment
Hide comment
@aronasorman
Member

aronasorman commented Mar 31, 2016

@aronasorman

This comment has been minimized.

Show comment
Hide comment
@aronasorman

aronasorman Mar 31, 2016

Member

@jamalex test failures seem related to your changes.

Member

aronasorman commented Mar 31, 2016

@jamalex test failures seem related to your changes.

@aronasorman

This comment has been minimized.

Show comment
Hide comment
@aronasorman

aronasorman Apr 1, 2016

Member

Tested with the sdist and en.zip; works as expected, it skipped the download prompt.

Member

aronasorman commented Apr 1, 2016

Tested with the sdist and en.zip; works as expected, it skipped the download prompt.

@aronasorman

This comment has been minimized.

Show comment
Hide comment
@aronasorman

aronasorman Apr 1, 2016

Member

I get this error with the sdist and khan_assessment.zip:

Step 5 : RUN kalite manage unpack_assessment_zip ./khan_assessment.zip
 ---> Running in 483a564cbe4e
Error reading secret key file. Generating one now. Error was: [Errno 2] No such file or directory: '/root/.kalite/secretkey.txt'
[WARNING] [2016-04-01 00:09:45,952] kalite: Channel khan does not exist.
[INFO] [2016-04-01 00:09:46,002] kalite: Unpacking...
...............[INFO] [2016-04-01 00:10:02,025] kalite: Scanning items and updating content db...
.[INFO] [2016-04-01 00:10:02,059] kalite: Annotating content for language: en, channel: khan
[ERROR] [2016-04-01 00:10:02,067] kalite: No content database file found
Traceback (most recent call last):
  File "/usr/share/kalite/python-packages/django/core/management/base.py", line 224, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/share/kalite/python-packages/django/core/management/base.py", line 263, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/dist-packages/kalite/contentload/management/commands/unpack_assessment_zip.py", line 81, in handle
    call_command("annotate_content_items")
  File "/usr/share/kalite/python-packages/django/core/management/__init__.py", line 161, in call_command
    return klass.execute(*args, **defaults)
  File "/usr/share/kalite/python-packages/django/core/management/base.py", line 263, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/dist-packages/kalite/topic_tools/management/commands/annotate_content_items.py", line 43, in handle
    annotate_content_models(database_path=database_path, channel=channel, language=language)
  File "/usr/local/lib/python2.7/dist-packages/kalite/topic_tools/content_models.py", line 128, in wrapper
    output = function(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/kalite/topic_tools/content_models.py", line 702, in annotate_content_models
    for path, update in content_models:
  File "/usr/local/lib/python2.7/dist-packages/kalite/topic_tools/content_models.py", line 607, in iterator_content_items
    items = Item.select().dicts().iterator()
  File "/usr/share/kalite/dist-packages/peewee.py", line 2797, in iterator
    return iter(self.execute().iterator())
  File "/usr/share/kalite/dist-packages/peewee.py", line 2787, in execute
    self._qr = ResultWrapper(model_class, self._execute(), query_meta)
  File "/usr/share/kalite/dist-packages/peewee.py", line 2470, in _execute
    return self.database.execute_sql(sql, params, self.require_commit)
  File "/usr/share/kalite/dist-packages/peewee.py", line 3199, in execute_sql
    self.commit()
  File "/usr/share/kalite/dist-packages/peewee.py", line 3048, in __exit__
    reraise(new_type, new_type(*exc_args), traceback)
  File "/usr/share/kalite/dist-packages/peewee.py", line 3191, in execute_sql
    cursor.execute(sql, params or ())
OperationalError: no such table: item
Member

aronasorman commented Apr 1, 2016

I get this error with the sdist and khan_assessment.zip:

Step 5 : RUN kalite manage unpack_assessment_zip ./khan_assessment.zip
 ---> Running in 483a564cbe4e
Error reading secret key file. Generating one now. Error was: [Errno 2] No such file or directory: '/root/.kalite/secretkey.txt'
[WARNING] [2016-04-01 00:09:45,952] kalite: Channel khan does not exist.
[INFO] [2016-04-01 00:09:46,002] kalite: Unpacking...
...............[INFO] [2016-04-01 00:10:02,025] kalite: Scanning items and updating content db...
.[INFO] [2016-04-01 00:10:02,059] kalite: Annotating content for language: en, channel: khan
[ERROR] [2016-04-01 00:10:02,067] kalite: No content database file found
Traceback (most recent call last):
  File "/usr/share/kalite/python-packages/django/core/management/base.py", line 224, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/share/kalite/python-packages/django/core/management/base.py", line 263, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/dist-packages/kalite/contentload/management/commands/unpack_assessment_zip.py", line 81, in handle
    call_command("annotate_content_items")
  File "/usr/share/kalite/python-packages/django/core/management/__init__.py", line 161, in call_command
    return klass.execute(*args, **defaults)
  File "/usr/share/kalite/python-packages/django/core/management/base.py", line 263, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/dist-packages/kalite/topic_tools/management/commands/annotate_content_items.py", line 43, in handle
    annotate_content_models(database_path=database_path, channel=channel, language=language)
  File "/usr/local/lib/python2.7/dist-packages/kalite/topic_tools/content_models.py", line 128, in wrapper
    output = function(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/kalite/topic_tools/content_models.py", line 702, in annotate_content_models
    for path, update in content_models:
  File "/usr/local/lib/python2.7/dist-packages/kalite/topic_tools/content_models.py", line 607, in iterator_content_items
    items = Item.select().dicts().iterator()
  File "/usr/share/kalite/dist-packages/peewee.py", line 2797, in iterator
    return iter(self.execute().iterator())
  File "/usr/share/kalite/dist-packages/peewee.py", line 2787, in execute
    self._qr = ResultWrapper(model_class, self._execute(), query_meta)
  File "/usr/share/kalite/dist-packages/peewee.py", line 2470, in _execute
    return self.database.execute_sql(sql, params, self.require_commit)
  File "/usr/share/kalite/dist-packages/peewee.py", line 3199, in execute_sql
    self.commit()
  File "/usr/share/kalite/dist-packages/peewee.py", line 3048, in __exit__
    reraise(new_type, new_type(*exc_args), traceback)
  File "/usr/share/kalite/dist-packages/peewee.py", line 3191, in execute_sql
    cursor.execute(sql, params or ())
OperationalError: no such table: item
@aronasorman

This comment has been minimized.

Show comment
Hide comment
@aronasorman

aronasorman Apr 1, 2016

Member

Adding --foreground does not solve the problem.

Seems like it's not finding the content.db. I thought we copied it to the right directory @benjaoming? Maybe not so for the sdist?

Member

aronasorman commented Apr 1, 2016

Adding --foreground does not solve the problem.

Seems like it's not finding the content.db. I thought we copied it to the right directory @benjaoming? Maybe not so for the sdist?

@aronasorman

This comment has been minimized.

Show comment
Hide comment
@aronasorman

aronasorman Apr 1, 2016

Member

For reference, here's the dockerfile I use to test this:

ROM gcr.io/google_appengine/python

COPY ka-lite*-static-*.tar.gz .
COPY khan_assessment.zip .
RUN pip install ka-lite*.tar.gz
RUN kalite manage unpack_assessment_zip ./khan_assessment.zip
ENTRYPOINT kalite manage setup
Member

aronasorman commented Apr 1, 2016

For reference, here's the dockerfile I use to test this:

ROM gcr.io/google_appengine/python

COPY ka-lite*-static-*.tar.gz .
COPY khan_assessment.zip .
RUN pip install ka-lite*.tar.gz
RUN kalite manage unpack_assessment_zip ./khan_assessment.zip
ENTRYPOINT kalite manage setup
@aronasorman

This comment has been minimized.

Show comment
Hide comment
@aronasorman

aronasorman Apr 1, 2016

Member

Working with @ralphiee22 to investigate this issue.

We tried to see where kalite is expecting the content db to be, then find out if it exists:

>>> os.path.exists('/root/.kalite/database/content_khan_en.sqlite')
False
>>> 
root@4fb0910d6a1b:/home/vmagent/app# find / -name content_khan_en.sqlite
/usr/local/lib/python2.7/dist-packages/kalite/database/templates/content_items/content_khan_en.sqlite
root@4fb0910d6a1b:/home/vmagent/app# 

Looks like it doesn't extract it anywhere at all. Expected behaviour @benjaoming? Is this the responsibility of an installer?

Member

aronasorman commented Apr 1, 2016

Working with @ralphiee22 to investigate this issue.

We tried to see where kalite is expecting the content db to be, then find out if it exists:

>>> os.path.exists('/root/.kalite/database/content_khan_en.sqlite')
False
>>> 
root@4fb0910d6a1b:/home/vmagent/app# find / -name content_khan_en.sqlite
/usr/local/lib/python2.7/dist-packages/kalite/database/templates/content_items/content_khan_en.sqlite
root@4fb0910d6a1b:/home/vmagent/app# 

Looks like it doesn't extract it anywhere at all. Expected behaviour @benjaoming? Is this the responsibility of an installer?

@aronasorman

This comment has been minimized.

Show comment
Hide comment
@aronasorman

aronasorman Apr 1, 2016

Member

Great success! After ignoring that error it successfully skips the content pack download prompt and continues on.

PR does what it intends to do. Once tests are fixed we can merge this.

Member

aronasorman commented Apr 1, 2016

Great success! After ignoring that error it successfully skips the content pack download prompt and continues on.

PR does what it intends to do. Once tests are fixed we can merge this.

@jamalex

This comment has been minimized.

Show comment
Hide comment
@jamalex

jamalex Apr 1, 2016

Member

@jamalex test failures seem related to your changes.

@aronasorman Can you point me to the specific failures you're referring to? I'm having trouble parsing out the test results. Only errors I see are:
[ERROR] [2016-03-31 21:47:16,122] kalite: Failed to end_user_activity upon logout: {'user': [u"Model facility user with pk u'39ce8862ffd94ce4a9ccc0d9df3bea64' does not exist."]} and some Selenium stuff, which don't seem related.

Member

jamalex commented Apr 1, 2016

@jamalex test failures seem related to your changes.

@aronasorman Can you point me to the specific failures you're referring to? I'm having trouble parsing out the test results. Only errors I see are:
[ERROR] [2016-03-31 21:47:16,122] kalite: Failed to end_user_activity upon logout: {'user': [u"Model facility user with pk u'39ce8862ffd94ce4a9ccc0d9df3bea64' does not exist."]} and some Selenium stuff, which don't seem related.

@aronasorman

This comment has been minimized.

Show comment
Hide comment
@aronasorman

aronasorman Apr 1, 2016

Member

Oh strange. Error from a previous run was about a topic not available. I'll
look closer.
On Mar 31, 2016 6:51 PM, "Jamie Alexandre" notifications@github.com wrote:

@jamalex https://github.com/jamalex test failures seem related to your
changes.

@aronasorman https://github.com/aronasorman Can you point me to the
specific failures you're referring to? I'm having trouble parsing out the
test results. Only errors I see are:
[ERROR] [2016-03-31 21:47:16,122] kalite: Failed to end_user_activity upon
logout: {'user': [u"Model facility user with pk
u'39ce8862ffd94ce4a9ccc0d9df3bea64' does not exist."]} and some Selenium
stuff, which don't seem related.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#5065 (comment)

Member

aronasorman commented Apr 1, 2016

Oh strange. Error from a previous run was about a topic not available. I'll
look closer.
On Mar 31, 2016 6:51 PM, "Jamie Alexandre" notifications@github.com wrote:

@jamalex https://github.com/jamalex test failures seem related to your
changes.

@aronasorman https://github.com/aronasorman Can you point me to the
specific failures you're referring to? I'm having trouble parsing out the
test results. Only errors I see are:
[ERROR] [2016-03-31 21:47:16,122] kalite: Failed to end_user_activity upon
logout: {'user': [u"Model facility user with pk
u'39ce8862ffd94ce4a9ccc0d9df3bea64' does not exist."]} and some Selenium
stuff, which don't seem related.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#5065 (comment)

@aronasorman

This comment has been minimized.

Show comment
Hide comment
@aronasorman

aronasorman Apr 1, 2016

Member

Failures are unrelated to changes. Succeeds on Jamie's PC. Merging so we can get 0.16.1 out.

Member

aronasorman commented Apr 1, 2016

Failures are unrelated to changes. Succeeds on Jamie's PC. Merging so we can get 0.16.1 out.

@aronasorman aronasorman merged commit dc399c9 into learningequality:0.16.x Apr 1, 2016

1 check failed

ci/circleci Your tests failed on CircleCI
Details

@aronasorman aronasorman deleted the jamalex:setup_contentpack_fixes branch Apr 1, 2016

@aronasorman aronasorman removed the has PR label Apr 1, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment