Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dependency: Pandas Version #1688

Merged
merged 6 commits into from
Oct 21, 2022
Merged

Dependency: Pandas Version #1688

merged 6 commits into from
Oct 21, 2022

Conversation

malachi-constant
Copy link
Contributor

Dependency Update

  • Pandas version >= 1.5.0 causing memory leak issues in some of our methods.

Detail

  • Locking version to <= 1.4.4 until resolved on the pandas side.

Relates

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@malachi-constant malachi-constant linked an issue Oct 14, 2022 that may be closed by this pull request
awswrangler.athena.read_sql_query is leaking memory #1678
Closed
@malachi-constant malachi-constant self-assigned this Oct 14, 2022
@malachi-constant malachi-constant added the dependencies Pull requests that update a dependency file label Oct 14, 2022
@malachi-constant
Copy link
Contributor Author

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: 0efd739
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jaidisido
Copy link
Contributor

Could we find or raise the actual issue causing this with pandas? I would want to avoid pinning the library until we understand why at least, specially that we need pandas 1.5.0+ for our 3.0.0 release

@malachi-constant
Copy link
Contributor Author

Could we find or raise the actual issue causing this with pandas? I would want to avoid pinning the library until we understand why at least, specially that we need pandas 1.5.0+ for our 3.0.0 release

Yep, I'm working on finding where memory leak happens using memory_profiler hopefully I'll know more after today.

@malachi-constant malachi-constant marked this pull request as draft October 17, 2022 15:55
@malachi-constant
Copy link
Contributor Author

Could we find or raise the actual issue causing this with pandas? I would want to avoid pinning the library until we understand why at least, specially that we need pandas 1.5.0+ for our 3.0.0 release

@jaidisido Confirmed this is no longer an issue with 1.5.1 users will need to be on python3.8 or newer.

@malachi-constant malachi-constant marked this pull request as ready for review October 21, 2022 00:18
@malachi-constant
Copy link
Contributor Author

malachi-constant commented Oct 21, 2022 •
edited

I'm not sure if this is the convention to exempt a dependency version for the library, let me know.

@malachi-constant
Copy link
Contributor Author

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: 9f76810
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@malachi-constant
Copy link
Contributor Author

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: 274ed6a
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jaidisido
Copy link
Contributor

jaidisido commented Oct 21, 2022 •
edited

Great deep dive on this one @malachi-constant , thanks!
Could you please add a comment linking to the issue in the pyproject so we can remember why we excluded this version?

@malachi-constant
Copy link
Contributor Author

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: fb96244
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@malachi-constant malachi-constant merged commit 3390360 into main Oct 21, 2022
@malachi-constant malachi-constant deleted the 1678-awswranglerathenaread_sql_query-is-leaking-memory branch October 21, 2022 18:28
@kukushking kukushking added this to the 2.18.0 milestone Dec 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@LeonLuttenberger LeonLuttenberger LeonLuttenberger approved these changes

@jaidisido jaidisido jaidisido approved these changes

@cnfait cnfait Awaiting requested review from cnfait

@kukushking kukushking Awaiting requested review from kukushking

Assignees

@malachi-constant malachi-constant

Labels
dependencies Pull requests that update a dependency file
Projects
None yet
Milestone
2.18.0
Development

Successfully merging this pull request may close these issues.

awswrangler.athena.read_sql_query is leaking memory
4 participants
@malachi-constant @jaidisido @LeonLuttenberger @kukushking