Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature 1436 storage and indexing #1695

Merged
merged 215 commits into from
Sep 28, 2023
Merged

Feature 1436 storage and indexing #1695

merged 215 commits into from
Sep 28, 2023

Conversation

taojing2002
Copy link
Contributor

This feature branch has two changes:

  1. Using dataone-indexer
  2. Removed Hazelcast

mbjones and others added 30 commits March 1, 2022 10:01
@mbjones
Development docs for storage and indexing refactor.
c80d5d2 
See issue #1436 and and #1364.
@mbjones
Merge branch 'develop' into feature-1436-storage-and-indexing
74967fd
@mbjones
Reformat POM.xml whitespace for consistency.
86a279b
@mbjones
Upgrade plantuml to current release jar.
74c793f
@mbjones
Initial pass at PlantUML component diagram.
9e0182e
@mbjones
Add plantuml jar to maven dependencies.
c28a0e1 
Currently unused by tooling -- need way to connect Makefile to use this jar.
@mbjones
Renegerated doc images with new plantuml version.
390be06
@mbjones
Second pass at component diagram.
6eb3bde 
Indexing components added, along with RabbitMQ and Ceph.
@mbjones
Indexing figure layout tweaks. First fairly complete draft.
e61228a
@mbjones
Ignore sphinx build artifacts.
ee83280
@gothub
Add indexing sequence diagrams
dd1855c 
Issue #1436 Add indexing sequence diagrams to storage subsystem document.
@mbjones
Remove extraneous vim swp file.
6620839
@mbjones
Add filenames to sequence diagrams so the generate in the correct loc…
52470a8 
…ation.
@taojing2002
Add two diagrams for the design document.
2dba8bf
@taojing2002
Update storage-subsystem.rst
5db336f
@mbjones
Changed proposed seq to show async calls to workers.
b37504c 
The actual worker process itself should also be asyn but is not shown that way currently.
@mbjones
Add basicAck to IndexWorker in proposed sequence diagram.
34b13c1
@mbjones
Rename classes, harmonize component diagram with sequence diagram.
bfb28f5
@mbjones
Remove hand-drawn sequence diagram that has been replaced.
c846d50
@taojing2002
Add the RabbitMQService class.
5466d25
@taojing2002
Use RabbitMQ to replace hazelcast on the index queue.
5e316b4
@taojing2002
Remove the list of fields from the index submit method. The list was …
8757d9d 
…only for the event log indexing, which is not used anymore.
@taojing2002
Rename the class of RabbitMQService to IndexGenerator.
321a85a
@taojing2002
Rename the package from rabbitmq to index.queue.
d917396
@taojing2002
Change the log statement for "RabbitMQService" to "IndexGenerator".
fee2d5f
@taojing2002
Remove system metadata from and add priority to the signature of the …
591ebd7 
…publish method.
@taojing2002
Add the method to get the file path.
49ab0fa 
Add a junit class for the IndexGenerator class.
@taojing2002
Add a log statement with the index task information such as the ident…
84a2ce3 
…ifier, index type and file path.
@taojing2002
Declare the max priority in the queue.
b370d1e
@taojing2002
Change the max priority to 10 for the rabbitmq.
4324bb2
artntek and others added 28 commits September 26, 2023 14:28
@artntek
delete commented code & wrap long lines
810d9b5
@artntek
rename updateSystemMetadataWithoutLock() -> updateSystemMetadata()
e041531
@taojing2002
Remove the properties related to hazelcast.
dbdddbd 
Remove the commented out rabitmq properties.
@artntek
del commented code. no whtspc changes
909a2ec
@taojing2002
Merge branch 'feature-1436-storage-and-indexing' of https://github.co…
14caf1e 
…m/nceas/metacat into feature-1436-storage-and-indexing
@artntek
whitespace only
cecbd01
@artntek
del commented code & wrap long lines
5950b35
@artntek
Merge branch 'feature-1436-storage-and-indexing' of github.com:NCEAS/…
107de17 
…metacat into feature-1436-storage-and-indexing
@artntek
whitespace only
691f25c
@artntek
del commented code & wrap long lines
56c7b23
@artntek
del commented code; wrap long lines; tabs to spaces
fa552c8
@artntek
minor edits
88c01ac
@artntek
whitespace only
c1eed7f
@artntek
del commented code; reformat; tabs -> spaces
4d9d4e2
@artntek
reformat long lines & remove licensing
b834d76
@artntek
whitespace only
b47d520
@artntek
del commented code; reformat long lines
88f8fb2
@artntek
remove licensing; wrap long line
080240a
@taojing2002
Add a comment to indicate the time regenerating index task use the de…
d98fdbf 
…fault time zone in the host.
@taojing2002
Reformat some long lines and correct some typos.
4076925
@taojing2002
Put + at the beginning of the lines.
d8e4f59
@taojing2002
Fixed a typo in the name of a constant.
fa01366
@taojing2002
Change the name of a constant since the definition was changed.
19b495e
@taojing2002
Make a variable final and wrap long lines.
96b2157
@taojing2002
Replace the thread sleeping for a long time by checking the solr quer…
4bdcec7 
…y result.
@taojing2002
Fixed an issue that a property was splitted into two strings.
0831c90 
Removed some commented out code.
@artntek
changed Angel Chen's ORCID to Jing's instead, since Angel no longer i…
64caace 
…n correct group
@artntek
added test-fast target
61cbe89
@taojing2002 taojing2002 merged commit 6e5b77b into develop Sep 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@artntek artntek artntek approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@taojing2002 @artntek @mbjones @gothub @doulikecookiedough