Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-13051][D] Enable pylint warnings (no-name-in-module/no-value-for-parameter) #16562

Conversation

roger-mike
Copy link
Contributor

@roger-mike roger-mike commented Jan 19, 2022

Fixes some no-value-for-parameter warnings.


Lang ULR Dataflow Flink Samza Spark Twister2
Go --- Build Status Build Status Build Status Build Status ---
Java Build Status Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Python --- Build Status
Build Status
Build Status
Build Status
Build Status
Build Status Build Status ---
XLang Build Status Build Status Build Status Build Status Build Status ---

Examples testing status on various runners

Lang ULR Dataflow Flink Samza Spark Twister2
Go --- --- --- --- --- --- ---
Java --- Build Status
Build Status
Build Status
--- --- --- --- ---
Python --- --- --- --- --- --- ---
XLang --- --- --- --- --- --- ---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go Java Python
Build Status Build Status Build Status
Build Status
Build Status

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website Whitespace Typescript
Non-portable Build Status
Build Status
Build Status
Build Status
Build Status
Build Status Build Status Build Status Build Status
Portable --- Build Status Build Status --- --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests

See CI.md for more information about GitHub Actions CI.

@codecov
Copy link

codecov bot commented Jan 19, 2022

Codecov Report

Merging #16562 (37acd46) into master (a589b10) will increase coverage by 37.19%.
The diff coverage is 50.00%.

Impacted file tree graph

@@             Coverage Diff             @@
##           master   #16562       +/-   ##
===========================================
+ Coverage   46.43%   83.63%   +37.19%     
===========================================
  Files         201      453      +252     
  Lines       19787    62562    +42775     
===========================================
+ Hits         9189    52322    +43133     
- Misses       9612    10240      +628     
+ Partials      986        0      -986     
Flag Coverage Δ
python 83.63% <50.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...beam/testing/load_tests/load_test_metrics_utils.py 34.45% <40.00%> (ø)
sdks/python/apache_beam/io/filesystem.py 89.41% <100.00%> (ø)
sdks/go/pkg/beam/artifact/stage.go
sdks/go/pkg/beam/core/runtime/graphx/user.go
sdks/go/pkg/beam/testing/passert/sum.go
sdks/go/pkg/beam/core/runtime/pipelinex/replace.go
sdks/go/pkg/beam/core/runtime/graphx/translate.go
sdks/go/pkg/beam/core/runtime/graphx/serialize.go
sdks/go/pkg/beam/runners/direct/gbk.go
sdks/go/pkg/beam/core/util/reflectx/tags.go
... and 646 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a589b10...37acd46. Read the comment docs.

@@ -3863,7 +3863,6 @@ def idxmin(self, **kwargs):
def idxmax(self, **kwargs):
return self._idxmaxmin_helper('idxmax', **kwargs)


Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated change

@@ -119,10 +119,8 @@ disable =
no-else-raise,
no-else-return,
no-member,
no-name-in-module,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The warnings here could be related to the 'import-error' and 'no-member' errors. Also not sure how to fix them.

@roger-mike
Copy link
Contributor Author

roger-mike commented Jan 21, 2022

The no-value-for-parameter warning could be fixed in some cases where there were some missing default values.

The rest of the no-value-for-parameter warnings are related to the @beam.ptransform_fn and frame_base.populate_defaults decorators. Not sure yet if these can be skipped or if a workaround exists to avoid the warning.

@roger-mike
Copy link
Contributor Author

R: @TheNeuralBit, could you take a look? Thanks

@TheNeuralBit TheNeuralBit self-requested a review January 22, 2022 00:37
@roger-mike
Copy link
Contributor Author

R: @pabloem could you take a look? Thanks.

Copy link
Member

@pabloem pabloem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -216,7 +216,7 @@ def __init__(
'InfluxDB')
self.filters = filters

def publish_metrics(self, result, extra_metrics: dict):
def publish_metrics(self, result, extra_metrics: dict = None):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you may need to tag the type as Optional[Dict]] so that this does not give an error.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done 👍

Comment on lines +100 to +104
directory=None,
prefix=None,
codec=None,
count=None,
sync_interval=None):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this necessary if the arguments are mandatory? Are you seeing a failure with this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they throw no-value-for-parameter warnings.

@roger-mike roger-mike force-pushed the fix/pylint/no-name-in-module-and-no-value-for-parameter branch from 6000dc1 to a65a755 Compare January 26, 2022 23:43
@roger-mike roger-mike force-pushed the fix/pylint/no-name-in-module-and-no-value-for-parameter branch from a65a755 to 495666e Compare January 27, 2022 21:58
@roger-mike
Copy link
Contributor Author

The no-value-for-parameter warning could be fixed in some cases where there were some missing default values.

The rest of the no-value-for-parameter warnings are related to the @beam.ptransform_fn and frame_base.populate_defaults decorators. Not sure yet if these can be skipped or if a workaround exists to avoid the warning.

@pabloem Do you have any pointers on this?

@roger-mike roger-mike changed the title [WIP][BEAM-13051][D] Enable pylint warnings (no-name-in-module/no-value-for-parameter) [BEAM-13051][D] Enable pylint warnings (no-name-in-module/no-value-for-parameter) Feb 14, 2022
@kileys
Copy link
Contributor

kileys commented Feb 16, 2022

@pabloem Could you take another look? Thanks!

@pabloem
Copy link
Member

pabloem commented Feb 24, 2022

sorrryyyy lookin

@pabloem
Copy link
Member

pabloem commented Feb 24, 2022

it looks like a lot of failures are occuring due to the newly enabled rules: https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Commit/12206/console

Can you fix those or re-enable the rule?

@roger-mike roger-mike force-pushed the fix/pylint/no-name-in-module-and-no-value-for-parameter branch from 495666e to f8428d2 Compare February 25, 2022 14:43
@roger-mike
Copy link
Contributor Author

Run Python PreCommit

it looks like a lot of failures are occuring due to the newly enabled rules: https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Commit/12206/console

Can you fix those or re-enable the rule?

I'll keep the rules enabled since all of them are false positives and fixing them won't add a lot of value.

@roger-mike
Copy link
Contributor Author

Run PythonLint PreCommit

2 similar comments
@roger-mike
Copy link
Contributor Author

Run PythonLint PreCommit

@roger-mike
Copy link
Contributor Author

Run PythonLint PreCommit

@roger-mike roger-mike force-pushed the fix/pylint/no-name-in-module-and-no-value-for-parameter branch 3 times, most recently from bd4ec61 to 5089d8c Compare March 2, 2022 20:04
@roger-mike roger-mike force-pushed the fix/pylint/no-name-in-module-and-no-value-for-parameter branch 2 times, most recently from df88746 to fa6a5d6 Compare March 2, 2022 20:45
@roger-mike roger-mike force-pushed the fix/pylint/no-name-in-module-and-no-value-for-parameter branch from fa6a5d6 to 74b0308 Compare March 2, 2022 20:52
@roger-mike roger-mike force-pushed the fix/pylint/no-name-in-module-and-no-value-for-parameter branch from 74b0308 to 37acd46 Compare March 2, 2022 21:27
@pabloem pabloem merged commit 46c1fc7 into apache:master Mar 2, 2022
@pabloem
Copy link
Member

pabloem commented Mar 2, 2022

lgtm thanks

nancyxu123 pushed a commit to nancyxu123/beam that referenced this pull request Mar 10, 2022
…rnings (no-name-in-module/no-value-for-parameter)

* [BEAM-13051] Pylint no-name-in-module and no-value-for-parameter warnings enabled

* [BEAM-13051] Fixed no-value-for-parameter warning for missing default values

* [BEAM-13051] Fixed parameters warnings
pabloem added a commit that referenced this pull request Mar 18, 2022
…r transaction boundaries and transaction ID ordering.

* Added integration test for transaction boundaries and transaction ID ordering. Made small fixes in ordered by key integration test.

* [BEAM-9150] Fix beam_PostRelease_Python_Candidate (python RC validation scripts) (#16955)

* Use default context output rather than outputWithTimestamp for ElasticsearchIO

* Palo Alto case study - fix link

* [BEAM-12777] Removed current docs version redirect

* Merge pull request #16850: [BEAM-11205] Upgrade Libraries BOM dependencies to 24.3.0

* Update GCP Libraries BOM version to 24.3.0

* Update associated dependencies

* Merge pull request #16484 from [BEAM-13633] [Playground] Implement method to get a default example for each SDKs

* Implement method to get a default example for each SDKs

* Add error handling

* Added saving of precompiled objects catalog to cache at the server startup

* Added caching of the catalog only in case of unspecified SDK

* Update regarding comments

* Update regarding comments

* Simplified logging regarding comment

* Get defaultExamplePath from the corresponding config

* Refactoring code

* Add the `link` field to response

* Remove gjson;
Resolve conflicts;

* Refactoring code

* Getting default precompiled object from cache

* Refactoring code

* Added saving of precompiled objects catalog to cache at the server startup

* Added caching of the catalog only in case of unspecified SDK

* Update regarding comments

* Update regarding comments

* Simplified logging regarding comment

* Updates regarding comments

* Update for environment_service_test.go

* Get default example from catalog

* GetCatalogFromCacheOrStorage method

* Update licenses

* Update licenses;
Resolve conflicts;

* [BEAM-13633][Playground]
Change saving default precompiled objects to the cache

* [BEAM-13633][Playground]
Change logic of saving and receiving info about default precompiled objects

* [BEAM-13633][Playground]
Separate for each sdk

* [BEAM-13633][Playground]
regenerate proto files

* Add code of the default example to response

* Revert "Add code of the default example to response"

This reverts commit da6baa0.

* Refactoring code

* Refactoring code;
Add test;

* Edit commentaries

* Refactoring code

* Add bucket name to methods

Co-authored-by: Artur Khanin <artur.khanin@akvelon.com>
Co-authored-by: AydarZaynutdinov <aydar.zaynutdinov@akvelon.com>
Co-authored-by: Pavel Avilov <pavel.avilov>

* Add 2022 events blog post (#16975)

* Clean up Go formatter suggestions (#16973)

* [BEAM-14012] Add go fmt to Github Actions (#16978)

* [BEAM-13911] Add basic tests to Go direct runner. (#16979)

* [BEAM-13960] Add support for more types when converting from between row and proto (#16875)

* Adding schema support.

* Addressing feedback.

* Bump org.mongodb:mongo-java-driver to 3.12.10

* [BEAM-13973] Link Dataproc Flink master URLs to the InteractiveRunner when FlinkRunner is used (#16904)

* [BEAM-13925] Turn pr bot on for go prs (#16984)

* [BEAM-13964] Bump kotlin to 1.6.x (#16882)

* [BEAM-13964] Bump kotlin to 1.6.x

* [BEAM-13964] Bump kotlin to 1.6.x

* [BEAM-13964] fix warnings in Kotlin compilation

* Skipping flaky sad-path tests for Spanner changestreams

* Merge pull request #16906: [BEAM-13974] Handle idle Storage Api streams

* Merge pull request #16562 from [BEAM-13051][D] Enable pylint warnings (no-name-in-module/no-value-for-parameter)

* [BEAM-13051] Pylint no-name-in-module and no-value-for-parameter warnings enabled

* [BEAM-13051] Fixed no-value-for-parameter warning for missing default values

* [BEAM-13051] Fixed parameters warnings

* [BEAM-13925] A couple small pr-bot bug fixes (#16996)

* [BEAM-14029] Add getter, setter for target maven repo (#16995)

* [BEAM-13903] Improve coverage of metricsx package (#16994)

* [BEAM-13892] Improve coverage of avroio package (#16990)

* [adhoc] Prepare aws2 ClientConfiguration for json serialization and cleanup AWS Module (#16894)

* [adhoc] Prepare aws2 ClientConfiguration and related classes for json serialization and cleanup AWS Module

* Merge pull request #16879 from [BEAM-12164] Add javadocs to SpannerConfig

* Add tests and config for retry

* lint

* add tests

* lint

* Delete tests not passing

* Rebase on apache beam master

* review changes

* review changes

* add javadocs to SpannerConfig

* revert

* add full stops

* [Cleanup] Update pre-v2 go package references (#17002)

* [BEAM-13885] Add unit tests to window package (#16971)

* Merge pull request #16891 from [BEAM-13872] [Playground] Increase test coverage for the code_processing package

* Increase test coverage for the code_processing package

* Refactoring code

* Add test cases with mock cache

* Add test for processCompileSuccess method

* Update test names

* Refactoring code

* Merge pull request #16912 from [BEAM-13878] [Playground] Increase test coverage for the fs_tool package

* Increase test coverage for the fs_tool package

* Rename folder

* Remove useless variable

* Update test names

* Merge pull request #16946 from [BEAM-13873] [Playground] Increase test coverage for the environment package

* Increase test coverage for the environment package

* Update test names

* Refactoring code

* Add bucket name to method

* [BEAM-13999] playground - support vertical orientation for graph

* [BEAM-13951] Update mass_comment.py list of Run commands (#16889)

* BEAM-13951: Sort run command list

* BEAM-13951: Update list

* fixup! BEAM-13951: Update list

* [BEAM-10652] Allow Clustering without Partition in BigQuery (#16578)

* [BEAM-10652] removed check that blocked clustering without partitioning

* [BEAM-10652] allow clustering without requiring partition

* newline

* added needed null

* remove testClusteringThrowsWithoutPartitioning

* update clustering

* formatting

* now compiles

* passes spotless

* update doc

* focus on single test

* spotless

* run all ITs

* spotless

* testing with time partitioning

* checking

* set clustering independant of partitioning

* remove timepart from it

* spotless

* removed test

* added TODO

* removed block of unneded code/comment

* remove override to v3 coder

* Spotless cleanup

* re-add override to v3 coder

* spotless

* adding checksum ( wrong value )

* added needed query var

* use tableName as var

* DATASET NAME

* project name in query

* update query

* change tests

* remove unneeded imports

* remove rest of forgotten

* add rows

* 16000 bytes

* bigint

* streaming test

* spotless

* methods

* end stream

* stream method and naming

* nostream

* streaming

* streamingoptions

* without streaming example

* string column instead of date -- related to BEAM-13753

* mor strings

* spotless

* revert, only DEFAULT and FILE_LOADS

* [BEAM-13857] Add K:V flags for expansion service jars and addresses to Go ITs. (#16908)

Adds functionality for running jars to the Go integration test framework, and uses this functionality to implement handling of K:V flags for providing expansion service jars and addresses to the test framework. This means that tests can simply get the address of an expansion service with the appropriate label, and this feature will handle running a jar if necessary, or just using the passed in endpoint otherwise.

* BEAM-14011 fix s3 filesystem multipart copy

* Merge pull request #16842 from [BEAM-13932][Playground] Container's user privileges

* [BEAM-13932][Playground]
Change Dockerfiles

* [BEAM-13932][Playground]
Update proxy and permissions for the container's user

* [BEAM-13932][Playground]
Update permissions for the container's user for scio

* Doc updates and blog post for 2.37.0 (#16887)

* Doc updates and blog post for 2.37.0

* Add BEAM-13980 to known issues

* Update dates

* Drop known issue (fix cherrypicked)

* Add license

* Add missing #

* Remove resolved issue in docs + update class path on sample (#17018)

* [BEAM-14016] Fixed flaky postcommit test (#17009)

Fixed SpannerWriteIntegrationTest.test_spanner_update by fixing the
metric exporter usage in spannerio.

* [BEAM-13925] months in date constructor are 0 indexed

* [BEAM-13947] Add split() and rsplit(), non-deferred column operations on categorical columns (#16677)

* Add split/rsplit; Need to refactor regex

* Support Regex; Refactor tests

* Remove debugger

* fix grammar

* Fix passing regex arg

* Reorder imports

* Address PR comments; Simplify kwargs

* Simplify getting columns for split_cat

* Update doctests to skip expand=True operations

* Fix missing doctest

* py: Import beam plugins before starting SdkHarness

* BEAM-14026 - Fixes bug related to Unnesting nested rows in an array (#16988)

* Suggested changes to handle nested row in an array

* Beam-14026 Suggested changes to handle nested row in an array

* Beam-14026 Enhanced by segregating the code from getBaseValues enhanced test case and example.

* Beam-14026 The code is moved from Row to avoid impact to the public interface.
The code is moved to BeamUnnestRel.java since its the caller class.
The Example code was duplicate, hence dropped.
build.gradle updated with the removal of example code.

* Remove resolved issue in notebook

* Bump numpy bound to include 1.22 and regenerate container deps.

* [BEAM-13925] Add ability to get metrics on pr-bot performance (#16985)

* Add script to get metrics on pr-bot performance

* Respond to feedback

* fix bad condition

* [BEAM-11085] Test that windows are correctly observed in DoFns

* Give pr bot write permissions on pr update

* Adding a logical type for Schemas using proto serialization. (#16940)

* BEAM-13765 missing PAssert methods (#16668)

* [BEAM-13909] improve coverage of Provision package (#17014)

* improve coverage of provision package

* updated comments

* [BEAM-14050] Update taxi.go example instructions

* Merge pull request #17027: [BEAM-11205] Upgrade GCP Libraries BOM dependencies to 24.4.0

* [BEAM-13709] Inconsistent behavior when parsing boolean flags across different APIs in Python SDK (#16929)

* Update dataflow API client.

* Instructions for updating apitools generated files.

* [BEAM-10976] Bundle finalization: Harness and some exec changes (#16980)

* Bundle finalization harness side changes

* Add testing

* Iterate over pardos directly

* Track bundlefinalizer in plan.go not pardo

* Remove outdated test

* Fix pointer issue

* Update todos to reference jiras

* Cleanup from feedback

* Doc nit

Co-authored-by: Daniel Oliveira <younghoono@gmail.com>

* GetExpirationTime comment

Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Daniel Oliveira <younghoono@gmail.com>

* Merge pull request #16976 from [BEAM-14010] [Website] Add Playground section to the Home page

* [BEAM-14010] [Website] Add Playground section to the Home page

* Update button to "Try Playground"

Co-authored-by: Aydar Zainutdinov <aydar.zaynutdinov@akvelon.com>

* [BEAM-14010] [Website] change button name

* [BEAM-14010] [Website] align header to center

* [BEAM-14010] [Website] change link

Co-authored-by: Alex Kosolapov <alex.kosolapov@akvelon.com>
Co-authored-by: Aydar Zainutdinov <aydar.zaynutdinov@akvelon.com>

* [BEAM-12447] Upgrade cloud build client and add/cleanup options (#17032)

* Merge pull request #17036 from [BEAM-12164] Convert all static instances to be transient in the connector in order to enable concurrent testing

* Convert all static instances to be transient in the connector in order to enable concurrent testing

* Initialized fields to null

* nullness

* Suppress uninitialized warnings

* Remove resetting dao factory fields in SpannerChangeStreamErrorTest.java

* Add validation package

* fix variable reference (#16991)

* Committed changes

* Print more logging

* More logging

* Made pipelines streaming

* Made small fixes

* Small fixes

* Ran spotless Apply

Co-authored-by: emily <emilyye@google.com>
Co-authored-by: egalpin <egalpin@users.noreply.github.com>
Co-authored-by: Aydar Farrakhov <stranniknm@gmail.com>
Co-authored-by: Miguel Hernandez <rogelio.hernandez@wizeline.com>
Co-authored-by: Benjamin Gonzalez <74670721+benWize@users.noreply.github.com>
Co-authored-by: Pavel Avilov <avilovpavel6@gmail.com>
Co-authored-by: Artur Khanin <artur.khanin@akvelon.com>
Co-authored-by: AydarZaynutdinov <aydar.zaynutdinov@akvelon.com>
Co-authored-by: Ahmet Altay <aaltay@gmail.com>
Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com>
Co-authored-by: Robert Burke <lostluck@users.noreply.github.com>
Co-authored-by: laraschmidt <laraschmidt@users.noreply.github.com>
Co-authored-by: Alexey Romanenko <aromanenko.dev@gmail.com>
Co-authored-by: Victor <career@victorplusc.com>
Co-authored-by: Danny McCormick <dannymccormick@google.com>
Co-authored-by: Masato Nakamura <randomstep@gmail.com>
Co-authored-by: Pablo Estrada <pabloem@apache.org>
Co-authored-by: reuvenlax <relax@google.com>
Co-authored-by: Miguel Hernandez <61989986+roger-mike@users.noreply.github.com>
Co-authored-by: Moritz Mack <mmack@talend.com>
Co-authored-by: Zoe <zoc@google.com>
Co-authored-by: Brian Hulette <bhulette@google.com>
Co-authored-by: brucearctor <5032356+brucearctor@users.noreply.github.com>
Co-authored-by: Daniel Oliveira <daniel.o.programmer@gmail.com>
Co-authored-by: sp029619 <stephen.patel@cerner.com>
Co-authored-by: David Cavazos <dcavazos@google.com>
Co-authored-by: Ning Kang <ningkang0957@gmail.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Andy Ye <andyye333@gmail.com>
Co-authored-by: Rahul Iyer <rahuliyer573@gmail.com>
Co-authored-by: abhijeet-lele <56114083+abhijeet-lele@users.noreply.github.com>
Co-authored-by: Valentyn Tymofieiev <valentyn@google.com>
Co-authored-by: Marcin Kuthan <marcin.kuthan@gmail.com>
Co-authored-by: Ritesh Ghorse <riteshghorse@gmail.com>
Co-authored-by: Jack McCluskey <jrmccluskey@google.com>
Co-authored-by: ansh0l <ansh0l@users.noreply.github.com>
Co-authored-by: Anand Inguva <34158215+AnandInguva@users.noreply.github.com>
Co-authored-by: Robert Bradshaw <robertwb@gmail.com>
Co-authored-by: Daniel Oliveira <younghoono@gmail.com>
Co-authored-by: bullet03 <bulat.safiullin@akvelon.com>
Co-authored-by: Alex Kosolapov <alex.kosolapov@akvelon.com>
Co-authored-by: Yichi Zhang <zyichi@google.com>
Co-authored-by: Nancy Xu <nancyxu@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants