New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Video upload (take 2) #929
Conversation
I have found a bug with this - It currently only allows support for chunked media uploads related to public content. As it turns out the media_category parameter is not differentiated solely by content type ("image", "video" or "gif") but by whether the content is public or private also. It can also take values of dm_image, dm_gif or dm_video. I will be looking into fixing this tomorrow, but would hesitate to merge this before it is addressed. |
It's been addressed, now what? |
At this point we are just waiting for a merge, we have been using this in production from the time I committed that last fix with no issues. |
Maybe you can merge the changes from the main repo into your fork? That
might give it a better chance of being merged.
…On Tue, Oct 17, 2017 at 4:12 AM, jamiewinspear ***@***.***> wrote:
At this point we are just waiting for a merge, we have been using this in
production from the time I committed that last fix with no issues.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#929 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AUIoiIVaYhxGXeg-VIAJexPTf8zZ3AhAks5stG9_gaJpZM4PfiXo>
.
|
@TheEssemCraft There aren't any conflicts between this branch and |
@jamesandres @jamiewinspear |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It works perfecty!!!
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
I just rebased on top of master and force-pushed. The only conflict was in a docstring. We have been using this in prod since 2017 and it seems fine. I will try to deploy this rebased version tomorrow. We have a little wrapper around it that shouldn't be contributed upstream, but is worth mentioning here:
|
cf54e31
to
d538993
Compare
Hi, any solution for this? My status updating on twitter profile without video, for the image it works fine. EDIT: It seems it's working fine but for some reasons, I'm getting "Not valid video" error. |
Any updates on the "Not valid video" error? Every video I try to upload seems to run into this. |
This comment has been minimized.
This comment has been minimized.
Hello there, i read a lot of issues here and, I try this
And I've got this message:
i don't know if I have an error or this is a problem with tweepy |
In Discord, it was asked to put together a MCVE for this so that's here now. This grabs an MP4 from the NHL back-end API and tries to post it to Twitter. It requires you to pass in consumer & access keys, but should be readily available - python3 nhl-highlight-ripper-poster.py --twitter \
--consumerkey "XXXXXXXX" --consumersecret "XXXXXXXX" \
--accesstoken "XXXXXXXX" --accesssecret "XXXXXXXX" import argparse
import requests
import shutil
import tweepy
parser = argparse.ArgumentParser()
parser.add_argument("--discord", help="log to console instead of file", action="store_true")
parser.add_argument("--disordwebhook", help="override team in configuration", action="store")
parser.add_argument("--twitter", help="log to console instead of file", action="store_true")
parser.add_argument("--consumerkey", help="override team in configuration", action="store")
parser.add_argument("--consumersecret", help="override team in configuration", action="store")
parser.add_argument("--accesstoken", help="override team in configuration", action="store")
parser.add_argument("--accesssecret", help="override team in configuration", action="store")
args = parser.parse_args()
def download_file(url):
local_filename = url.split("/")[-1]
with requests.get(url, stream=True) as r:
with open(local_filename, "wb") as f:
shutil.copyfileobj(r.raw, f)
return local_filename
sample_video = download_file(
"https://hlslive-wsczoominwestus.med.nhl.com/publish/80b062f2-4778-4224-8dbb-df201e7eb233.mp4"
)
payload = {"content": "NHL highlight video."}
video_file = open(sample_video, "rb")
files = {"file": video_file}
if args.discord:
response = requests.post(args.discordwebhook, files=files, data=payload)
if args.twitter:
consumer_key = args.consumerkey
consumer_secret = args.consumersecret
access_token = args.accesstoken
access_secret = args.accesssecret
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_secret)
api = tweepy.API(auth)
media_ids = [api.media_upload(sample_video).media_id_string]
status = api.update_status(status="NHL highlight video.", media_ids = media_ids)
if not response.ok:
print(response.json()) This is the error returned from code -
Here is the
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The interaction between upload_chunked
and _chunk_media
is messy.
_chunk_media
should be split into three separate methods, corresponding to INIT, APPEND, and FINALIZE.
They don't need to be public methods, but each should be a bound API method so that the logic in upload_chunked
can be refactored.
There are also conflicts now that need to be resolved.
max_size_standard = 5120 # standard uploads must be less then 5 MB | ||
max_size_chunked = 15360 # chunked uploads must be less than 15 MB | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a reason these aren't initialized in __init__
or better yet, constants outside the class?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems to be 5 and 15 MiB rather than MB. Have these limits been tested?
The library is currently using limits of 4883 and 14649 KiB right now, corresponding to just over 5 and 15 MB.
from tweepy.utils import list_to_csv | ||
|
||
IMAGE_MIMETYPES = ('image/gif', 'image/jpeg', 'image/png', 'image/webp') | ||
CHUNKED_MIMETYPES = ('image/gif', 'image/jpeg', 'image/png', 'image/webp', 'video/mp4') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CHUNKED_MIMETYPES = ('image/gif', 'image/jpeg', 'image/png', 'image/webp', 'video/mp4') | |
CHUNKED_MIMETYPES = IMAGE_MIMETYPES + ('video/mp4',) |
""" | ||
f = kwargs.pop('file', None) | ||
|
||
mime, _ = mimetypes.guess_type(filename) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This has been changed to use imghdr.what
with #1086.
mimetypes.guess_type
should probably only be used as a fallback in the case that imghdr.what
fails to determine the type to check if the file is an mp4.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
imghdr
doesn't work on video files, so that's another reason to keep it as a backup.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
size = f.tell() | ||
f.seek(0) | ||
|
||
if mime in IMAGE_MIMETYPES and size < self.max_size_standard: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
os.path.getsize
returns the size in bytes.
Given a filename, this will chunk any image greater than 5120 bytes rather than 5 MB.
raise TweepError("Can't upload media with mime type %s" % mime) | ||
|
||
def image_upload(self, filename, *args, **kwargs): | ||
""" :reference: https://dev.twitter.com/rest/reference/post/media/upload |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
""" :reference: https://dev.twitter.com/rest/reference/post/media/upload | |
""" :reference: https://developer.twitter.com/en/docs/media/upload-media/api-reference/post-media-upload |
try: | ||
if file_size > (max_size * 1024): | ||
raise TweepError('File is too big, must be less than %skb.' % max_size) | ||
except os.error as e: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
except os.error as e: | |
except OSError as e: |
except os.error as e: | ||
raise TweepError('Unable to access file: %s' % e.strerror) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
os.path.getsize
is outside the try
block and nothing else in it will raise OSError
.
raise TweepError('File is too big, must be less than %skb.' % max_size) | ||
f.seek(0) # Reset to beginning of file | ||
fp = f | ||
elif command != 'finalize': |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The only other option is
elif command != 'finalize': | |
elif command == 'append': |
# video must be mp4 | ||
file_type, _ = mimetypes.guess_type(filename) | ||
|
||
if file_type is None: | ||
raise TweepError('Could not determine file type') | ||
|
||
if file_type not in CHUNKED_MIMETYPES: | ||
raise TweepError('Invalid file type for video: %s' % file_type) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was already determined in media_upload
and could be an image.
The file type should be passed to this method rather than redetermined here.
if command == 'init': | ||
query = { | ||
'command': 'INIT', | ||
'media_type': file_type, | ||
'total_bytes': file_size, | ||
'media_category': API._get_media_category( | ||
is_direct_message, file_type) | ||
} | ||
body.append(urlencode(query).encode('utf-8')) | ||
headers = { | ||
'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8' | ||
} | ||
elif command == 'append': | ||
if media_id is None: | ||
raise TweepError('Media ID is required for APPEND command.') | ||
body.append(b'--' + BOUNDARY) | ||
body.append('Content-Disposition: form-data; name="command"'.encode('utf-8')) | ||
body.append(b'') | ||
body.append(b'APPEND') | ||
body.append(b'--' + BOUNDARY) | ||
body.append('Content-Disposition: form-data; name="media_id"'.encode('utf-8')) | ||
body.append(b'') | ||
body.append(str(media_id).encode('utf-8')) | ||
body.append(b'--' + BOUNDARY) | ||
body.append('Content-Disposition: form-data; name="segment_index"'.encode('utf-8')) | ||
body.append(b'') | ||
body.append(str(segment_index).encode('utf-8')) | ||
body.append(b'--' + BOUNDARY) | ||
body.append('Content-Disposition: form-data; name="{0}"; filename="{1}"'.format(form_field, os.path.basename(filename)).encode('utf-8')) | ||
body.append('Content-Type: {0}'.format(file_type).encode('utf-8')) | ||
body.append(b'') | ||
body.append(fp.read(chunk_size)) | ||
body.append(b'--' + BOUNDARY + b'--') | ||
headers = { | ||
'Content-Type': 'multipart/form-data; boundary=Tw3ePy' | ||
} | ||
elif command == 'finalize': | ||
if media_id is None: | ||
raise TweepError('Media ID is required for FINALIZE command.') | ||
body.append( | ||
urlencode({ | ||
'command': 'FINALIZE', | ||
'media_id': media_id | ||
}).encode('utf-8') | ||
) | ||
headers = { | ||
'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8' | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This can be combined with the prior if-elif statement.
@jamesandres @jamiewinspear @alsuren I've gone ahead and done a first pass at a review of the PR.
There also seem to be further improvements by @fitnr that haven't been included in this PR yet. @braian87b @jh69 @corbindavenport @Spyder-exe @iandow @mjarrett You can simply install from this PR or the branch of the fork this PR is from rather than creating or using another fork with the same changes. @umar13893 @awcarlsson Do your videos meet the criteria in the Video specifications and recommendations section of https://developer.twitter.com/en/docs/media/upload-media/uploading-media/media-best-practices? @Shumaister @mattdonders Did you install Tweepy from this PR? |
Is this safe to pull to use prior to the v4 launch? Also, sort of of new to patching libraries.. I installed Tweepy using |
There shouldn't be any risk to using this PR, but there are still issues that need to be resolved. See https://tweepy.readthedocs.io/en/latest/install.html and https://pip.pypa.io/en/stable/reference/pip_install/#git. |
Superseded by #1414 |
The video-upload branch / pull request #1486 should be complete now. |
This is a more up to date version of #672 originally crafted by @Choko256 and @fitnr.
The original description from the previous PR: