Draft: IDIOMS Update & BULKI v0.1 #203

zhangwei217245 · 2024-06-24T03:40:06Z

Related Issues / Pull Requests

Direct
#151 #169 #206

Indirect
#109 #132

Description

Major changes:

IDIOMS -> affix-based query benchmark
IDIOMS -> index persistence mechanism
IDIOMS -> numeric value query support (exact query and range query)
IDIOMS -> numeric value query benchmark
IDIOMS -> simulation test
BULKI V0.1 -> a data serialization and deserialization mechanism
Metadata JSON validator
Metadata JSON Loader for PDC

Minor changes

Updated Dockerfile for faster docker image build
Updated pdc_generic.h for full data type support capability and better ordering of PDC data type enums

In-progress changes

Some code for multi-condition query.

What changes are proposed in this pull request?

Bug fix (non-breaking change which fixes an issue)
[ x ] New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality not to work as expected; for instance, examples in this repository must be updated too)
[ x ] This change requires a documentation update

Checklist:

My code modifies existing public API, or introduces new public API, and I updated or wrote docstrings
[ x ] I have commented my code
My code requires documentation updates, and I have made corresponding changes to the documentation
[ x ] I have added tests that prove my fix is effective or that my feature works
[ x ] New and existing unit tests pass locally with my changes

Merge branch 'develop' of github.com:zhangwei217245/pdc into feature/dart

…dart

Major changes: * IDIOMS -> affix-based query benchmark * IDIOMS -> Simulation Test * IDIOMS -> Multi data type supported for the same key * IDIOMS -> Range Query and Exact Query for Numeric Values * IDIOMS -> benchmark for numeric values (exact search and range query) * IDIOMS -> Index Persistence * BULKI -> A data serialization and deserialization mechanism.

src/api/pdc_client_connect.c

jeanbez · 2024-07-01T16:26:54Z

src/api/pdc_client_connect.c

        HG_Destroy(dart_get_server_info_handle);
-        return ret;
+        return;


Just confirming: why do we not need a return code anymore?

The previous implementation was taking serverID and returning a dart_server struct instance that contains the indexed_word_count as well as request_count fields. However, when writing the simulator test of IDIOMS, I realize that we can send a dart_server pointer as an input, which can pass in the server ID, while still be able to take the result information of the function. This simplifies the implementation of , not only the simulator test, but also everywhere else in IDIOMS.

jeanbez · 2024-07-01T16:29:07Z

src/commons/generic/include/pdc_generic_old.h

@@ -0,0 +1,264 @@
+#ifndef PDC_GENERIC_H


Do we need this _old.h file?

since all the existing test passed, I don't think we need this anymore.

Got it! Could you remove this file, then? If needed we can get it back for git

Sure, let's remove it. I will update this PR along with the end-to-end CI test of IDIOMS.

jeanbez · 2024-07-01T16:30:05Z

src/commons/index/dart/index/hashtable/hashtable_impl.c

@@ -0,0 +1,174 @@
+


This entire file commented out, do we still need it or can we remove it?

This is for the controlled experiment in the DART paper (for full-string hash and prefix hashing). But now, this is not used.
Let's leave this here for this version. Will possibly need it somewhere in the future.

jeanbez · 2024-07-01T16:30:24Z

src/commons/index/dart/index/hashtable/include/hashtable_impl.h

@@ -0,0 +1,25 @@
+


Same for this file, is it still needed?

This is for the controlled experiment in the DART paper (for full-string hash and prefix hashing). But now, this is not used.
Let's leave this here for this version. Will possibly need it somewhere in the future.

src/server/include/pdc_server.h

jeanbez · 2024-07-01T17:10:18Z

src/server/include/pdc_server_metadata.h

@@ -32,7 +32,7 @@
 #include "mercury_proc_string.h"
 #include "mercury_atomic.h"

-#include "pdc_hash-table.h"


The header file seems to be removed in the PR, but it is still included. Is this correct?

the "pdc_hash-table.h" is different than "pdc_hash_table.h" in terms of the file name and location. But both files contain the same implementation. After this change, we only have one location for the pdc_hash_table which is located in pdc_commons. This helps with avoid redundant code as well as inconsistency between two copies of the exact same code.

jeanbez · 2024-07-01T17:12:23Z

CMakeLists.txt

+  set(SUPPRESSED_LIST "-Wno-alloc-size-larger-than" ${SUPPRESSED_LIST})
+  set(SUPPRESSED_LIST "-Wno-array-bounds" ${SUPPRESSED_LIST})
+  set(SUPPRESSED_LIST "-Wno-stringop-truncation" ${SUPPRESSED_LIST})
+  set(SUPPRESSED_LIST "-Wno-use-after-free" ${SUPPRESSED_LIST})


Why are we suppressing those?

these are some warnings that can be ignored. Given the complexity of our PDC implementation, these warnings are not very sense-making in our project and there is marginal value for fixing these warnings in the code.

We need a cleaner compilation log to highlight the compilation errors that really matters.

-Wno-use-after-free should probably be removed from here, is there an example code segment and compiler warning message that shows it is not helping?

src/commons/index/dart/index/idioms/idioms_local_index_test.c

jeanbez · 2024-07-01T17:15:03Z

src/commons/logging/pdc_logger.c

DIdn't we have a logger in PDC already? Is this used only for IDIOMS?

AFAIK, there is no logger that controls different level of logging.

zhangwei217245 and others added 30 commits October 13, 2023 15:58

address conflict

b32bc76

address conflict

52ab0d7

address conflict

ec47fe3

address conflict

154aa14

updating scripts

1feacf4

Merge branch 'stable' into develop

4f79787

Merge branch 'develop' into develop

bc36933

updating scripts

d8f962c

Merge branch 'develop' of github.com:zhangwei217245/pdc into develop

c7a55d7

:wq

f63dc22

Merge branch 'develop' of github.com:zhangwei217245/pdc into feature/dart

update container config

0183f1f

Merge branch 'develop' of github.com:zhangwei217245/pdc into feature/…

48a370f

…dart

fix cmake mercury_util not found issue

ec6c6bc

update for Julia support

f99a537

fix hdf5.h not found for src/tools

0751a12

update container config

c446108

update container config

9911445

update container config

8296539

update container config

d1303da

update container config

88a75e6

update container config

923ec93

update container config

9c3fbc4

update container config

2f2c073

update container config

49ef572

update container config

199c1bb

add libhdf5-dev for Github Actions

7fc75f0

Merge branch 'develop' of github.com:zhangwei217245/pdc into feature/…

05511d7

…dart

update CMake for HDF5 in tools

11c8cec

update logic for finding HDF5

71a23d5

update

5e01ca9

zhangwei217245 and others added 10 commits May 28, 2024 17:24

remove old kvtag benchmarks

bb3ee9f

update

f7e12a6

update

7085ea4

update changes

861ab85

resolve conflict

57987f6

Merge branch 'hpc-io:develop' into develop

5575907

dart info

6b146bc

update

377f20c

multi data type for the same key, supported now

6ad301f

zhangwei217245 self-assigned this Jun 24, 2024

zhangwei217245 temporarily deployed to external June 24, 2024 03:40 — with GitHub Actions Inactive

zhangwei217245 added 2 commits June 24, 2024 14:06

fix CMakeLists.txt

05c35cf

update

b14b26b

zhangwei217245 requested a deployment to external June 24, 2024 14:09 — with GitHub Actions Waiting

update

9a14049

zhangwei217245 requested a deployment to external June 24, 2024 14:26 — with GitHub Actions Waiting

update format

bdaede9

zhangwei217245 requested a deployment to external June 24, 2024 14:41 — with GitHub Actions Waiting

zhangwei217245 added 2 commits June 24, 2024 15:10

update BULKI interface order

9df1b3c

BULKI API sorted

abd329b

zhangwei217245 deployed to external June 24, 2024 16:38 — with GitHub Actions Active

zhangwei217245 requested review from jeanbez, houjun and sbyna June 24, 2024 17:50

jeanbez requested changes Jul 1, 2024

View reviewed changes

jeanbez added type: enhancement New feature or request type: new feature Request for new feature labels Jul 1, 2024

jeanbez added this to the v.0.6 milestone Jul 1, 2024

jeanbez changed the title ~~IDIOMS Update & BULKI v0.1~~ Draft: IDIOMS Update & BULKI v0.1 Jul 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Draft: IDIOMS Update & BULKI v0.1 #203

Draft: IDIOMS Update & BULKI v0.1 #203

zhangwei217245 commented Jun 24, 2024 •

edited

Loading

jeanbez Jul 1, 2024

zhangwei217245 Jul 2, 2024

jeanbez Jul 1, 2024

zhangwei217245 Jul 2, 2024

jeanbez Jul 2, 2024

zhangwei217245 Jul 2, 2024 •

edited

Loading

jeanbez Jul 1, 2024

zhangwei217245 Jul 2, 2024

jeanbez Jul 1, 2024

zhangwei217245 Jul 2, 2024

jeanbez Jul 1, 2024

zhangwei217245 Jul 2, 2024

jeanbez Jul 1, 2024

zhangwei217245 Jul 2, 2024

houjun Jul 2, 2024

jeanbez Jul 1, 2024

zhangwei217245 Jul 2, 2024

Draft: IDIOMS Update & BULKI v0.1 #203

Are you sure you want to change the base?

Draft: IDIOMS Update & BULKI v0.1 #203

Conversation

zhangwei217245 commented Jun 24, 2024 • edited Loading

Related Issues / Pull Requests

Description

What changes are proposed in this pull request?

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhangwei217245 Jul 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhangwei217245 commented Jun 24, 2024 •

edited

Loading

zhangwei217245 Jul 2, 2024 •

edited

Loading