1508 avro schemas #60
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 30 commits
April 13, 2021 13:21
…mentation) to take dictionary with extra parameters allowed as well as methods within this class not to reference config but the variable that was set. Adjusted effected tests.
…WebPublisher_unit as unit test until create folder structure for integration vs unit tests.
…o take dictionary with extra parameters allowed as well as methods within this class not to reference config but the variable that was set. Adjusted effected tests.
… to take dictionary with extra parameters allowed as well as methods within this class not to reference config but the variable that was set. Adjusted effected tests. Removed get_logger method as it wasn't used and we used a different logger then. Added checks for if security wasn't enabled.
… using a topic variable instead. Removed changing name of the consumer_conf to meta_consumer_conf. Added additional logging.
… clearer how similar code is to KafkaPublisher class.
…ould notice this usage).
…) to take dictionary with extra parameters allowed as well as methods within this class not to reference config but the variable that was set. Removed logging import and changed to ClientLogger.
…es sense. Added test for extra arguments via constructor. Fixed test for testing less parameters passed in, because more could have made it through.
…rn values a little better where could, and changed a test of assertTrue to assertEqual so got to see what the actual value was when failed.
…olidate generating the key from UUID, and added a little logging. Consolidated topic generation code too.
… file_id_prefix as it is in half the other places including S3MessageAdapter constructor.
…tra parameters allowed. Refactored out of SqsConsumer the connecting part and put into S3Utils, this left only log_level as class var. Put creating a Queue object into receive_messages (can refactor out if ever need again, but single line didn't seem to warrent its own method). Added debug logging.
…arameters for receive_message so a user could create their own queue or use our connect to do so and pass either in.
… with extra parameters allowed so adjusted the tests and added more with more verification.
…usted SqsConsumer callback parameters to pass along log_level.
added 28 commits
July 1, 2021 21:05
…il class methods.
…il class, didn't need so much setup. Moved constructor test up with other constructor test.
…ercase.Changed RelationshipType to be Enum class like RecordType, to reflect schemas repo.
…id intellij separated the changes.
…et defined, config_dict wasn't defined yet.
…e that's all it is transforming on. Renamed var in test_S3MessageAdapter to be more general name.
… is operating on.
…ma_registry, which is a Map/Dict {str, FileLocation}. Also fixed sme script for last change of FileLocationType being an enum, not class with var type.
…stry. Not a blank class.
…f schemas not psiSchemaClasses.
…lass name in the assertion.
…rameter as a record. This is because our S3MessageAdapter.transform, SqsHandler methods, and SqsConsumer.receive_messages all eventually only operate on 1 record.
…(value['content'], object_hook=as_enum)
…anged handler in script to work off the actual value passed in not just the test data in there.
…types. Removed some duplicated tests (ones covered by other tests).
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.