Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add a warning about text length? #20

Open
erdnaxel opened this issue Feb 4, 2024 · 0 comments
Open

add a warning about text length? #20

erdnaxel opened this issue Feb 4, 2024 · 0 comments

Comments

@erdnaxel
Copy link

erdnaxel commented Feb 4, 2024

Hello: I've been using your package to get the sentiment of tweets. It's been working very well, so thank you!

Recently, the data source I use to retrieve tweets has started providing the full text of tweets, instead of a truncated version when the tweets are long. (Some tweets are very, very long.)

Not knowing better, I continued to analyze tweet sentiment as before, but I started getting tensorflow warning messages about using more than 10% of memory. After these messages, the model would just stall and never finish the job.

I tried running the sentiment_score() function on smaller batches of tweets and that worked for a few batches, but then the problem arose again for a single small batch.

I knew the number of tweets definitely wasn't the problem -- I was only doing 100 at a time -- so I figured the length of the text must be the issue. I added a step to truncate tweets to 400 characters and that completely solved the problem.

My suggestion would be to add a warning about the length of the text affecting the performance in the Troubleshooting section of the page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant