-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cadc-11115 new caom2-meta-sync module #247
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Haven't looked at Main yet; config/doc changes first and that will reduce what's in Main
…om caom2harvester, updated caom2-meta-sync to use a config file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
README and configuration:
Until we get a response from Job on the db, let's simplify as much as possible:
- get rid of dryrun - it's useless
- change runContinuously to exitWhenComplete, make it optional, and default to false: that way run forever is the normal mode and this is an exception for special cases
- both
threads
andbatchSize
are implementation details that we unfortunately expose to operators and explaining sensible values will be very hard... let's make those internal and for now hard code to batchSize=100 and nthreads=batchSize/10. - change maxSleep to maxIdle and keep it with the "source" config items
- hide (remove?) the "source db" config documentation for now
- use
org.opencadc.caom2.metasync.repoService
for source config, move theorg.opencadc.caom2.metasync.collection
up beside the repoService and make it clear that multiple values are allowed; put all "source:" config before "destination" config - use
org.opencadc.caom2.metasync.db.*
for destination config; keeporg.opencadc.caom2.metasync.basePublisherID
with destination config
Style: In the text that explains each config property, you don't need to write out the whole property. It's hard to read... see fenwick (trackSiteLocations
) or tantar (buckets) for two styles that seem to be sufficient ref to the config spec and more readable (just inconsistent, but I think I prefer the italics since they aren't literally exact).
No description provided.