Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: encode the csv data before save #100

Merged
merged 1 commit into from
Aug 15, 2022

Conversation

arslanashraf7
Copy link
Contributor

@arslanashraf7 arslanashraf7 commented Apr 13, 2022

Description: Describe in a couple of sentences what this PR adds
This PR adds python's default(utf-8) encoding on the file data before saving.
This is not needed in the case of local storage while manipulating the grades but it becomes an issue when we use AWS/S3 storage as a backend instead of the local storage. The s3 services throw errors about file encoding.

A sample error log is attached below.

JIRA: Link to JIRA ticket
More details can be seen on mitodl/mitxonline#415

Dependencies: dependencies on other outstanding PRs, issues, etc.
None

Merge deadline: List merge deadline (if any)
No Specific Date but Soon

Installation instructions: List any non-trivial installation
instructions.

Testing instructions:
Mentioned in mitodl/mitxonline#415

Sample Error Log
The sample error log can be seen:

2022-04-11 21:05:44,654 ERROR 7292 [root] [user None] [ip None] signals.py:22 - Uncaught exception from None
Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/bulk_grades/views.py", line 45, in dispatch
    return super().dispatch(request, course_id, *args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/bulk_grades/views.py", line 98, in post
    self.processor.process_file(the_file, autocommit=True)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/super_csv/csv_processor.py", line 188, in process_file
    self.preprocess_file(reader)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/bulk_grades/api.py", line 355, in preprocess_file
    self.save()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/bulk_grades/api.py", line 333, in save
    return super().save(operating_user=self._user)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/super_csv/mixins.py", line 118, in save
    operation = CSVOperation.record_operation(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/super_csv/models.py", line 72, in record_operation
    instance.data.save(uuid.uuid4(), ContentFile(data))
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/db/models/fields/files.py", line 89, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/core/files/storage.py", line 54, in save
    name = self._save(name, content)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 495, in _save
    self._save_content(obj, content, parameters=parameters)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 510, in _save_content
    obj.upload_fileobj(content, ExtraArgs=put_parameters)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/boto3/s3/inject.py", line 511, in object_upload_fileobj
    return self.meta.client.upload_fileobj(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
    return future.result()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/client.py", line 591, in _make_api_call
    handler, event_response = self.meta.events.emit_until_response(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/hooks.py", line 242, in emit_until_response
    responses = self._emit(event_name, kwargs, stop_on_response=True)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/handlers.py", line 209, in conditionally_calculate_md5
    calculate_md5(params, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/handlers.py", line 187, in calculate_md5
    binary_md5 = _calculate_md5_from_file(body)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/handlers.py", line 201, in _calculate_md5_from_file
    md5.update(chunk)
TypeError: Unicode-objects must be encoded before hashing

Reviewers:

  • tag reviewer
  • tag reviewer

Merge checklist:

  • All reviewers approved
  • CI build is green
  • Version bumped
  • Changelog record added
  • Documentation updated (not only docstrings)
  • Commits are squashed

Post merge:

  • Create a tag
  • Check new version is pushed to PyPI after tag-triggered build is
    finished.
  • Delete working branch (if not needed anymore)

Author concerns: List any concerns about this PR - inelegant
solutions, hacks, quick-and-dirty implementations, concerns about
migrations, etc.

@openedx-webhooks openedx-webhooks added needs triage open-source-contribution PR author is not from Axim or 2U labels Apr 13, 2022
@openedx-webhooks
Copy link

openedx-webhooks commented Apr 13, 2022

Thanks for the pull request, @arslanashraf7! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

  • supporting documentation
  • Open edX discussion forum threads
  • timeline information ("this must be merged by XX date", and why that is)
  • partner information ("this is a course on edx.org")
  • any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

@arslanashraf7 arslanashraf7 changed the title fix: encode the csv data before saving fix: encode the csv data before save Apr 13, 2022
@natabene
Copy link

@arslanashraf7 Thank you for your contribution. Is this ready for our review?

@arslanashraf7
Copy link
Contributor Author

@natabene Yes, it's ready for review.

Copy link
Contributor

@leangseu-edx leangseu-edx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@leangseu-edx leangseu-edx merged commit 786a4f8 into openedx:master Aug 15, 2022
@openedx-webhooks
Copy link

@arslanashraf7 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
open-source-contribution PR author is not from Axim or 2U
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

5 participants