Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Use requests to fetch event and speaker images #6639

Merged
merged 1 commit into from Dec 6, 2019

Conversation

codedsun
Copy link
Contributor

@codedsun codedsun commented Dec 5, 2019

Fixes #6618

Short description of what this resolves:

Resolves image-resizing task which was throwing error 403

Checklist

  • I have read the Contribution & Best practices Guide and my PR follows them.
  • My branch is up-to-date with the Upstream development branch.
  • The unit tests pass locally with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)
  • All the functions created/modified in this PR contain relevant docstrings.

@auto-label auto-label bot added the fix label Dec 5, 2019
@codedsun
Copy link
Contributor Author

codedsun commented Dec 5, 2019

@iamareebjamal The test fails and gives Corrupt/Invalid images for few

python-pillow/Pillow#3287

For now, the image resizing is working with local images as well as postman

@codedsun
Copy link
Contributor Author

codedsun commented Dec 5, 2019

Traceback (most recent call last):
  File "/Users/codedsun/Documents/FOSS/open-event-server/app/api/helpers/files.py", line 85, in create_save_resized_image
    im = Image.open(image_file)
  File "/usr/local/lib/python3.7/site-packages/PIL/Image.py", line 2818, in open
    raise IOError("cannot identify image file %r" % (filename if filename else fp))
OSError: cannot identify image file <_io.BytesIO object at 0x11266bc50>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/flask_celeryext/app.py", line 101, in __call__
    res = Task.__call__(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/sentry_sdk/integrations/celery.py", line 158, in _inner
    reraise(*exc_info)
  File "/usr/local/lib/python3.7/site-packages/sentry_sdk/_compat.py", line 57, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/sentry_sdk/integrations/celery.py", line 153, in _inner
    return f(*args, **kwargs)
  File "/Users/codedsun/Documents/FOSS/open-event-server/app/api/helpers/tasks.py", line 147, in resize_user_images_task
    uploaded_images = create_save_image_sizes(original_image_url, 'speaker-image', user.id)
  File "/Users/codedsun/Documents/FOSS/open-event-server/app/api/helpers/files.py", line 154, in create_save_image_sizes
    'original_image_url': create_save_resized_image(image_file, 0, 0, 0, original_upload_path, resize=False),
  File "/Users/codedsun/Documents/FOSS/open-event-server/app/api/helpers/files.py", line 87, in create_save_resized_image
    raise IOError("Corrupt/Invalid Image")
OSError: Corrupt/Invalid Image

Log

Also, tried ImageFile.LOAD_TRUNCATED_IMAGES = True. It's not working for the create-user api in documentation where we used this image https://cdn.pixabay.com/photo/2013/11/23/16/25/birds-216412_1280.jpg

@iamareebjamal
Copy link
Member

That's because r.raw.read() is empty.

It's simply Image.open(io.BytesIO(response.content))

@codecov
Copy link

codecov bot commented Dec 5, 2019

Codecov Report

Merging #6639 into development will increase coverage by <.01%.
The diff coverage is 100%.

Impacted file tree graph

@@               Coverage Diff               @@
##           development    #6639      +/-   ##
===============================================
+ Coverage        65.35%   65.35%   +<.01%     
===============================================
  Files              298      298              
  Lines            15239    15240       +1     
===============================================
+ Hits              9959     9960       +1     
  Misses            5280     5280
Impacted Files Coverage Δ
app/api/helpers/files.py 65.6% <100%> (+0.22%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6d098cb...aaf929f. Read the comment docs.


import PIL
from PIL import Image
from PIL import Image, ImageFile

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'PIL.ImageFile' imported but unused

@iamareebjamal iamareebjamal changed the title fix: Image resizing performed using requests library fix: Image fetch performed using requests library Dec 5, 2019
@@ -77,7 +78,7 @@ def create_save_resized_image(image_file, basewidth=None, maintain_aspect=None,
if not image_file:
return None
filename = '{filename}.{ext}'.format(filename=get_file_name(), ext=ext)
data = urllib.request.urlopen(image_file).read()
data = requests.get(image_file, stream=True).content
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why stream=True?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was previously using raw, have updated.

@iamareebjamal
Copy link
Member

dredd tests are broken. Fix them

@codedsun
Copy link
Contributor Author

codedsun commented Dec 5, 2019

@iamareebjamal Check now!

@iamareebjamal
Copy link
Member

iamareebjamal commented Dec 5, 2019

Please don't just skip the transactions, just delete the entire dredd tests as well. Why test anything, right?

@codedsun
Copy link
Contributor Author

codedsun commented Dec 5, 2019

Sorry, I didn't mean that.

Let me dive into dredd, I m new to it. I dont' know what is causing the response to be 500, when the actual response from postman is 201. Let me troubleshoot it again! I will take time and fix it. Thanks

@iamareebjamal
Copy link
Member

It's simple example.com/example.png does not exist

@iamareebjamal
Copy link
Member

INFO:werkzeug:127.0.0.1 - - [05/Dec/2019 21:07:46] "PATCH /v1/users/2 HTTP/1.1" 500 -
ERROR:werkzeug:Error on request:
Traceback (most recent call last):
  File "/home/circleci/code/venv/lib/python3.7/site-packages/werkzeug/serving.py", line 304, in run_wsgi
    execute(self.server.app)
  File "/home/circleci/code/venv/lib/python3.7/site-packages/werkzeug/serving.py", line 292, in execute
    application_iter = app(environ, start_response)
  File "/home/circleci/code/venv/lib/python3.7/site-packages/sentry_sdk/integrations/flask.py", line 70, in sentry_patched_wsgi_app
    environ, start_response
  File "/home/circleci/code/venv/lib/python3.7/site-packages/sentry_sdk/integrations/wsgi.py", line 120, in __call__
    reraise(*_capture_exception(hub))
  File "/home/circleci/code/venv/lib/python3.7/site-packages/sentry_sdk/_compat.py", line 57, in reraise
    raise value
  File "/home/circleci/code/venv/lib/python3.7/site-packages/sentry_sdk/integrations/wsgi.py", line 116, in __call__
    _sentry_start_response, start_response, span
  File "/home/circleci/code/venv/lib/python3.7/site-packages/sentry_sdk/integrations/flask.py", line 69, in <lambda>
    return SentryWsgiMiddleware(lambda *a, **kw: old_app(self, *a, **kw))(
  File "/home/circleci/code/venv/lib/python3.7/site-packages/flask/app.py", line 2463, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/circleci/code/app/__init__.py", line 77, in __call__
    return self.app(environ, start_response)
  File "/home/circleci/code/venv/lib/python3.7/site-packages/flask/app.py", line 2449, in wsgi_app
    response = self.handle_exception(e)
  File "/home/circleci/code/venv/lib/python3.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/circleci/code/venv/lib/python3.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/circleci/code/venv/lib/python3.7/site-packages/flask/app.py", line 1866, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/circleci/code/venv/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/circleci/code/venv/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/circleci/code/venv/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/circleci/code/venv/lib/python3.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/circleci/code/venv/lib/python3.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/circleci/code/venv/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/circleci/code/venv/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/circleci/code/venv/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/circleci/code/venv/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/circleci/code/venv/src/flask-rest-jsonapi/flask_rest_jsonapi/api.py", line 133, in decorated
    return self.check_permissions(view, view_args, view_kwargs, *args, **kwargs)
  File "/home/circleci/code/app/api/helpers/permission_manager.py", line 488, in permission_manager
    return permissions[args[0]](view, view_args, view_kwargs, *args, **kwargs)
  File "/home/circleci/code/app/api/helpers/permissions.py", line 46, in decorator
    return fn(*args, **kwargs)
  File "/home/circleci/code/app/api/helpers/permission_manager.py", line 127, in is_user_itself
    return view(*view_args, **view_kwargs)
  File "/home/circleci/code/venv/src/flask-rest-jsonapi/flask_rest_jsonapi/decorators.py", line 32, in wrapper
    return func(*args, **kwargs)
  File "/home/circleci/code/venv/lib/python3.7/site-packages/flask/views.py", line 89, in view
    return self.dispatch_request(*args, **kwargs)
  File "/home/circleci/code/venv/src/flask-rest-jsonapi/flask_rest_jsonapi/resource.py", line 68, in dispatch_request
    response = method(*args, **kwargs)
  File "/home/circleci/code/venv/src/flask-rest-jsonapi/flask_rest_jsonapi/decorators.py", line 56, in wrapper
    return func(*args, **kwargs)
  File "/home/circleci/code/venv/src/flask-rest-jsonapi/flask_rest_jsonapi/resource.py", line 310, in patch
    self._data_layer.update_object(obj, data, kwargs)
  File "/home/circleci/code/venv/src/flask-rest-jsonapi/flask_rest_jsonapi/data_layers/alchemy.py", line 126, in update_object
    self.before_update_object(obj, data, view_kwargs)
  File "/home/circleci/code/app/api/users.py", line 266, in before_update_object
    start_image_resizing_tasks(user, data['avatar_url'])
  File "/home/circleci/code/app/api/users.py", line 326, in start_image_resizing_tasks
    resize_user_images_task.delay(user_id, original_image_url)
  File "/home/circleci/code/venv/lib/python3.7/site-packages/celery/app/task.py", line 427, in delay
    return self.apply_async(args, kwargs)
  File "/home/circleci/code/venv/lib/python3.7/site-packages/sentry_sdk/integrations/celery.py", line 97, in apply_async
    return f(*args, **kwargs)
  File "/home/circleci/code/venv/lib/python3.7/site-packages/celery/app/task.py", line 552, in apply_async
    link=link, link_error=link_error, **options)
  File "/home/circleci/code/venv/lib/python3.7/site-packages/celery/app/task.py", line 774, in apply
    ret = tracer(task_id, args, kwargs, request)
  File "/home/circleci/code/venv/lib/python3.7/site-packages/sentry_sdk/integrations/celery.py", line 135, in _inner
    return f(*args, **kwargs)
  File "/home/circleci/code/venv/lib/python3.7/site-packages/celery/app/trace.py", line 399, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "/home/circleci/code/venv/lib/python3.7/site-packages/celery/app/trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/circleci/code/venv/lib/python3.7/site-packages/flask_celeryext/app.py", line 101, in __call__
    res = Task.__call__(self, *args, **kwargs)
  File "/home/circleci/code/venv/lib/python3.7/site-packages/celery/app/task.py", line 394, in __call__
    return self.run(*args, **kwargs)
  File "/home/circleci/code/venv/lib/python3.7/site-packages/sentry_sdk/integrations/celery.py", line 158, in _inner
    reraise(*exc_info)
  File "/home/circleci/code/venv/lib/python3.7/site-packages/sentry_sdk/_compat.py", line 57, in reraise
    raise value
  File "/home/circleci/code/venv/lib/python3.7/site-packages/sentry_sdk/integrations/celery.py", line 153, in _inner
    return f(*args, **kwargs)
  File "/home/circleci/code/app/api/helpers/tasks.py", line 147, in resize_user_images_task
    uploaded_images = create_save_image_sizes(original_image_url, 'speaker-image', user.id)
  File "/home/circleci/code/app/api/helpers/files.py", line 155, in create_save_image_sizes
    small_upload_path),
  File "/home/circleci/code/app/api/helpers/files.py", line 96, in create_save_resized_image
    width_percent = (basewidth / float(img.size[0]))
TypeError: unsupported operand type(s) for /: 'NoneType' and 'float'

@codedsun
Copy link
Contributor Author

codedsun commented Dec 6, 2019

@iamareebjamal What shall be done for this issue

  width_percent = (basewidth / float(img.size[0]))
TypeError: unsupported operand type(s) for /: 'NoneType' and 'float'

I can understand that basewidth is becoming none, but when we are calling the function we are passing the basewidth for every image size, is it somwhere we are passing none to the function means image basewidth is coming to be None?

@iamareebjamal
Copy link
Member

That, you have to debug as to why it is becoming None

@codedsun
Copy link
Contributor Author

codedsun commented Dec 6, 2019

@iamareebjamal Okay debugging, taking time for it.

@codedsun codedsun force-pushed the image-fix branch 3 times, most recently from 00dc952 to 06b0dfc Compare December 6, 2019 13:07
@@ -77,6 +78,7 @@ def create_save_resized_image(image_file, basewidth=None, maintain_aspect=None,
if not image_file:
return None
filename = '{filename}.{ext}'.format(filename=get_file_name(), ext=ext)
# data = requests.get(image_file).content
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? Debug things locally. Don't push intermediate changes

@codedsun
Copy link
Contributor Author

codedsun commented Dec 6, 2019

@iamareebjamal I logged locally all the basewidth sizes, they are coming to be appropriate sizes and working fine for both the API's. I will take more time to debug, I don't know what is getting wrong.

@codedsun
Copy link
Contributor Author

codedsun commented Dec 6, 2019

@iamareebjamal Pheew!!!. This has passed all the test, please check.
Thanks for your patience and bearing it up with me. This has taken long and provided a very good learning for me. Also exact 24 hrs :P

icon_width=75, icon_height=30, icon_aspect=True,
icon_quality=80, thumbnail_width=500, thumbnail_height=200,
thumbnail_aspect=True, thumbnail_quality=80, logo_width=500,
logo_height=200)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for condition, add all values unconditionally

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The values are different for speaker-type and event-type

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, they are mutually exclusive

@@ -213,7 +213,7 @@ Create a new user using an email, password and an optional name.
"twitter-url": "http://twitter.com/twitter",
"instagram-url": "http://instagram.com/instagram",
"google-plus-url": "http://plus.google.com/plus.google",
"original-image-url": "https://cdn.pixabay.com/photo/2013/11/23/16/25/birds-216412_1280.jpg"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to change this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

"icon-image-url": "http://example.com/example.png"
"thumbnail-image-url": "https://image.flaticon.com/icons/png/64/126/126486.png",
"small-image-url": "https://image.flaticon.com/icons/png/64/126/126486.png",
"icon-image-url": "https://image.flaticon.com/icons/png/64/126/126486.png"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are read only and should be removed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove these fields or revert the change? Removing from the documentation response will be like the documentation is incomplete? Your call, tell me what

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, they are discarded. So, the documentation will be complete

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iamareebjamal Test are failing because the request becomes incomplete

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, they are failing because it's invalid JSON

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reverting this change to the previous one.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can't have trailing commas in JSON

@codedsun codedsun force-pushed the image-fix branch 2 times, most recently from e25eca1 to dbd71c0 Compare December 6, 2019 19:24
app/api/helpers/files.py Outdated Show resolved Hide resolved
app/api/helpers/files.py Outdated Show resolved Hide resolved
app/api/helpers/files.py Outdated Show resolved Hide resolved
app/api/helpers/files.py Outdated Show resolved Hide resolved
app/api/helpers/files.py Outdated Show resolved Hide resolved
app/api/helpers/files.py Outdated Show resolved Hide resolved
app/api/helpers/files.py Outdated Show resolved Hide resolved
@@ -559,9 +559,6 @@ Authorized user should be same as user in request body or must be admin.
"twitter-url": "http://twitter.com/twitter",
"instagram-url": "http://instagram.com/instagram",
"google-plus-url": "http://plus.google.com/plus.google",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JSON can't have trailing commas

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was a blunder, apologies.

@niranjan94
Copy link
Member

Codacy Here is an overview of what got changed by this pull request:

Clones added
============
- app/api/helpers/files.py  1
         

See the complete overview on Codacy

@codedsun
Copy link
Contributor Author

codedsun commented Dec 6, 2019

Check now!

@iamareebjamal
Copy link
Member

Great work. Now, work on creating a management task which resizes the images of events and speakers which were not done previously

@iamareebjamal iamareebjamal changed the title fix: Image fetch performed using requests library fix: Use requests to fetch event and speaker images Dec 6, 2019
@iamareebjamal iamareebjamal merged commit e8b367f into fossasia:development Dec 6, 2019
@codedsun
Copy link
Contributor Author

codedsun commented Dec 6, 2019

This wasn't great as I used to do, it had lot of blunders from my side. Thnkyou for the help and efforts 👍 May be my bad health made it more worse but yes I did even in this case ^_^

Now, work on creating a management task which resizes the images of events and speakers which were not done previously

New Issue for this?

@iamareebjamal
Copy link
Member

No, follow up to this PR as it was a part of this issue only

@codedsun
Copy link
Contributor Author

codedsun commented Dec 6, 2019

Can you give me any example of management-task written in the repo or tell me what it's similar to as in java-servers.

No, follow up to this PR as it was a part of this issue only

Okay, will send a new PR with the same issue tag.

@iamareebjamal
Copy link
Member

manage.py

@codedsun
Copy link
Contributor Author

codedsun commented Dec 7, 2019

@iamareebjamal I have created the management command but directly invoking the function of create_resized image, the urllib gives connection refused error. Do I need to invoke the same celery task?

@iamareebjamal
Copy link
Member

urllib gives connection refused error

You replaced urllib with requests yourself

@codedsun
Copy link
Contributor Author

codedsun commented Dec 7, 2019

@iamareebjamal Yes, sorry I fixed that. My server was not running hence the error was coming up. Also do i need to create seperate management commands for events and speakers or one? Thanks

@codedsun
Copy link
Contributor Author

codedsun commented Dec 7, 2019

urllib gives connection refused error

You replaced urllib with requests yourself

Yes and facing and fixing the dredd tests was fantastic tasks :P

@iamareebjamal
Copy link
Member

One

codedsun added a commit to codedsun/open-event-server that referenced this pull request Dec 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Image resizing is not working
4 participants