1507 update scripts #58
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 30 commits
April 13, 2021 13:21
…mentation) to take dictionary with extra parameters allowed as well as methods within this class not to reference config but the variable that was set. Adjusted effected tests.
…WebPublisher_unit as unit test until create folder structure for integration vs unit tests.
…o take dictionary with extra parameters allowed as well as methods within this class not to reference config but the variable that was set. Adjusted effected tests.
… to take dictionary with extra parameters allowed as well as methods within this class not to reference config but the variable that was set. Adjusted effected tests. Removed get_logger method as it wasn't used and we used a different logger then. Added checks for if security wasn't enabled.
… using a topic variable instead. Removed changing name of the consumer_conf to meta_consumer_conf. Added additional logging.
… clearer how similar code is to KafkaPublisher class.
…ould notice this usage).
…) to take dictionary with extra parameters allowed as well as methods within this class not to reference config but the variable that was set. Removed logging import and changed to ClientLogger.
…es sense. Added test for extra arguments via constructor. Fixed test for testing less parameters passed in, because more could have made it through.
…rn values a little better where could, and changed a test of assertTrue to assertEqual so got to see what the actual value was when failed.
…olidate generating the key from UUID, and added a little logging. Consolidated topic generation code too.
… file_id_prefix as it is in half the other places including S3MessageAdapter constructor.
…tra parameters allowed. Refactored out of SqsConsumer the connecting part and put into S3Utils, this left only log_level as class var. Put creating a Queue object into receive_messages (can refactor out if ever need again, but single line didn't seem to warrent its own method). Added debug logging.
…arameters for receive_message so a user could create their own queue or use our connect to do so and pass either in.
… with extra parameters allowed so adjusted the tests and added more with more verification.
…usted SqsConsumer callback parameters to pass along log_level.
added 29 commits
June 3, 2021 22:13
…d if it metadata_type wasn't GRANULE or COLLECTION. Changed try/raise in consume method to try/finally with same closing of the consumer as before. This way exceptions do fail script, because before was unclear why it wasn't working.
…Name was a Delete type.
…ention in last commit that the AvroDeserializer constructor needed more specific parameter names (think it got updated and our order of params was wrong).
…dict instead of config location. Removed mocking for launch_delete_handler script since have unit tests that cover this now.
…er of params seem to have changed.
…er of params seem to have changed. (forgot intellij's multi select for commits doesn't work)
…so as to get better error reporting.
…on so as to get better error reporting.
…to what they are on cedar-devs.
…was changed a while ago in the code).
…pdated test to reflect/catch.
…ilds with old code and hoping is because it was being built before the onestop-python-client-build.
…require onestop-python-client-test to run and the two script builds to require onestop-python-client-build to run.
… build requires configuration.
…d do a json dumps. Was complaining ParsedRecord doesn't have method serialize. Fixed tests, also added size and versionid to test util message since S3MessageAdapter.transform required.
… to os*, is url on cedardevs.
…Adjusted some log statements.
…ode shouldn't be able to throw any exceptions, the ones that could happen you want to bubble up (like connection problems). Moved the closing of connection after while polling section, since was closing the connection.
… there's a problem with the callback it gets thrown - also the sqs_message.delete is outside try so if a connection problem that ends everything.
…not serialize the json payload but instead do a json dumps.
…dict instead of config location.
…tement since no harm in extra params.
…_metadata_type, including changing type var used in S3MessageAdapter.
… methods. Mostly added, removed a redundant one.Adjusted one in deliver reports from error to info.
…nformation, and tried to organize by the automated process vs manual. Second pass to come in next story.
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #1507