blockSize - a new config paramter for max size of the block #272

navinrathore · 2022-05-18T11:59:37Z

No description provided.

sonalgoyal · 2022-05-19T14:55:30Z

client/src/main/java/zingg/client/Arguments.java

@@ -622,7 +623,15 @@ public boolean getShowConcise() {
 	public void setShowConcise(boolean showConcise) {
 		this.showConcise = showConcise;
 	}
-
+
+	public long getblockSize() {


camel casing is missing - getBlockSize, same for setter

this will break the json parsing - please also test one end to end case with 120l records and set block size in args. Use debug logs to verify

Changed to Camel Case.
Ran with examples/febrl120k/config.json. Output attached. Appropriate block size was selected.

sonalgoyal · 2022-05-19T14:57:36Z

core/src/main/java/zingg/util/Heuristics.java

-
-	public static long getMaxBlockSize(long totalCount) {
+	public static final long MIN_SIZE = 8L;
+	public static long getMaxBlockSize(long totalCount, long blockSizeFromConfig) {
 		long maxSize = 8;


shouldnt this be set to the min size var ?

Used MIN_SIZE

sonalgoyal · 2022-05-19T14:59:46Z

core/src/main/java/zingg/util/Heuristics.java

-			LOG.debug("**Block size found **" + maxSize);
-			if (maxSize > 100) maxSize = 100;
+			LOG.debug("**Block size found **");
+			if (maxSize > blockSizeFromConfig) maxSize = blockSizeFromConfig;
 			if (maxSize <= 8) maxSize = 8;


use the defined constant

used MIN_SIZE

navinrathore · 2022-05-23T09:31:21Z

Output:
examples/febrl120k/config.json

A. ==================================================
Phase: findTrainingData
blockSize(Config): 150

 2022-05-23 12:38:27,107 [main] INFO  zingg.util.Heuristics - **Block size **12 and total count was 12123
 2022-05-23 12:38:27,121 [main] INFO  zingg.util.Heuristics - Heuristics suggest 12

B ==================================================
Phase: findTrainingData
blockSize(Config): 10

 2022-05-23 13:56:30,606 [main] WARN  zingg.TrainingDataFinder - Read training samples 37 neg 43
 2022-05-23 13:56:31,093 [main] INFO  zingg.TrainingDataFinder - Preprocessing DS for stopWords
 2022-05-23 13:56:50,121 [main] INFO  zingg.util.Heuristics - **Block size **10 and total count was 12035
 2022-05-23 13:56:50,121 [main] INFO  zingg.util.Heuristics - Heuristics suggest 10
 2022-05-23 13:56:50,121 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 10
 2022-05-23 13:57:00,114 [main] WARN  org.apache.spark.sql.execution.CacheManager - Asked to cache already cached
 
 C ==================================================
Phase: findTrainingData
blockSize(Config): 5
 
  2022-05-23 13:59:39,839 [main] INFO  zingg.TrainingDataFinder - Preprocessing DS for stopWords
 2022-05-23 13:59:58,061 [main] INFO  zingg.util.Heuristics - **Block size **8 and total count was 11993
 2022-05-23 13:59:58,061 [main] INFO  zingg.util.Heuristics - Heuristics suggest 8
 2022-05-23 13:59:58,061 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 8
 
D ==================================================
Phase: Train
blockSize(Config): 5

 2022-05-23 14:03:44,657 [main] INFO  zingg.util.Heuristics - **Block size **8 and total count was 30406
 2022-05-23 14:03:44,658 [main] INFO  zingg.util.Heuristics - Heuristics suggest 8
 2022-05-23 14:03:44,658 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 8
 2022-05-23 14:03:59,095 [main] WARN  zingg.util.PipeUtil - Writing output Pipe [name=null, format=PARQUET, preprocessors=null, props={location=models/101/model/block/zingg.block}, schema=null]
 2022-05-23 14:03:59,095 [main] WARN  zingg.util.PipeUtil - Writing file

E ======================================================
Phase: Train
blockSize(Config): 50

ingType,true))]
 2022-05-23 14:08:30,628 [main] INFO  zingg.util.Heuristics - **Block size **30 and total count was 30186
 2022-05-23 14:08:30,629 [main] INFO  zingg.util.Heuristics - Heuristics suggest 30
 2022-05-23 14:08:30,629 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 30
 2022-05-23 14:08:43,103 [main] WARN  zingg.util.PipeUtil - Writing output Pipe [name=null, format=PARQUET, preprocessors=null, props={location=models/101/model/block/zingg.block}, schema=null]
 2022-05-23 14:08:43,104 [main] WARN  zingg.util.PipeUtil - Writing file
F ======================================================
Phase: Train
blockSize(Config): 25

 2022-05-23 14:11:17,582 [main] INFO  zingg.util.Heuristics - **Block size **25 and total count was 30061
 2022-05-23 14:11:17,582 [main] INFO  zingg.util.Heuristics - Heuristics suggest 25
 2022-05-23 14:11:17,583 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 25
======================================================

navinrathore · 2022-05-23T09:55:29Z

Output:
examples/febrl120k/config.json

A. ==================================================
Phase: findTrainingData
blockSize(Config): 150

 2022-05-23 12:38:27,107 [main] INFO  zingg.util.Heuristics - **Block size **12 and total count was 12123
 2022-05-23 12:38:27,121 [main] INFO  zingg.util.Heuristics - Heuristics suggest 12

B ==================================================
Phase: findTrainingData
blockSize(Config): 10

 2022-05-23 13:56:30,606 [main] WARN  zingg.TrainingDataFinder - Read training samples 37 neg 43
 2022-05-23 13:56:31,093 [main] INFO  zingg.TrainingDataFinder - Preprocessing DS for stopWords
 2022-05-23 13:56:50,121 [main] INFO  zingg.util.Heuristics - **Block size **10 and total count was 12035
 2022-05-23 13:56:50,121 [main] INFO  zingg.util.Heuristics - Heuristics suggest 10
 2022-05-23 13:56:50,121 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 10
 2022-05-23 13:57:00,114 [main] WARN  org.apache.spark.sql.execution.CacheManager - Asked to cache already cached
 
 C ==================================================
Phase: findTrainingData
blockSize(Config): 5
 
  2022-05-23 13:59:39,839 [main] INFO  zingg.TrainingDataFinder - Preprocessing DS for stopWords
 2022-05-23 13:59:58,061 [main] INFO  zingg.util.Heuristics - **Block size **8 and total count was 11993
 2022-05-23 13:59:58,061 [main] INFO  zingg.util.Heuristics - Heuristics suggest 8
 2022-05-23 13:59:58,061 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 8
 
D ==================================================
Phase: Train
blockSize(Config): 5

 2022-05-23 14:03:44,657 [main] INFO  zingg.util.Heuristics - **Block size **8 and total count was 30406
 2022-05-23 14:03:44,658 [main] INFO  zingg.util.Heuristics - Heuristics suggest 8
 2022-05-23 14:03:44,658 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 8
 2022-05-23 14:03:59,095 [main] WARN  zingg.util.PipeUtil - Writing output Pipe [name=null, format=PARQUET, preprocessors=null, props={location=models/101/model/block/zingg.block}, schema=null]
 2022-05-23 14:03:59,095 [main] WARN  zingg.util.PipeUtil - Writing file

E ======================================================
Phase: Train
blockSize(Config): 50

ingType,true))]
 2022-05-23 14:08:30,628 [main] INFO  zingg.util.Heuristics - **Block size **30 and total count was 30186
 2022-05-23 14:08:30,629 [main] INFO  zingg.util.Heuristics - Heuristics suggest 30
 2022-05-23 14:08:30,629 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 30
 2022-05-23 14:08:43,103 [main] WARN  zingg.util.PipeUtil - Writing output Pipe [name=null, format=PARQUET, preprocessors=null, props={location=models/101/model/block/zingg.block}, schema=null]
 2022-05-23 14:08:43,104 [main] WARN  zingg.util.PipeUtil - Writing file
F ======================================================
Phase: Train
blockSize(Config): 25

 2022-05-23 14:11:17,582 [main] INFO  zingg.util.Heuristics - **Block size **25 and total count was 30061
 2022-05-23 14:11:17,582 [main] INFO  zingg.util.Heuristics - Heuristics suggest 25
 2022-05-23 14:11:17,583 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 25
======================================================

sonalgoyal · 2022-05-23T09:26:29Z

core/src/main/java/zingg/util/Heuristics.java

 		long maxSize = 8;
 		/*if  (totalCount > 100 && totalCount < 500){
 			maxSize = totalCount / 5;
 		}
 		else {*/	
 			maxSize = (long) (0.001 * totalCount);
-			LOG.debug("**Block size found **" + maxSize);
-			if (maxSize > 100) maxSize = 100;
+			LOG.debug("**Block size found **");


please print max size here

navinrathore added 3 commits May 18, 2022 17:28

blockSize - a new config paramter for max size of the block

0317c7f

var renamed to blockSize

4ecadfb

var renamed to blockSize - adding missed files

dbb5555

sonalgoyal requested changes May 19, 2022

View reviewed changes

Merge branch 'zinggAI:main' into BlockingTreeSize259

88b5533

changed blockSize to camelCase in setter/getter

3df49f1

sonalgoyal requested changes May 23, 2022

View reviewed changes

restored maxSize logging

7c9a327

sonalgoyal merged commit b4d6fee into zinggAI:main May 23, 2022

sonalgoyal mentioned this pull request May 23, 2022

expose blocking tree size as a parameter #259

Closed

navinrathore deleted the BlockingTreeSize259 branch June 1, 2022 04:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

blockSize - a new config paramter for max size of the block #272

blockSize - a new config paramter for max size of the block #272

navinrathore commented May 18, 2022

sonalgoyal May 19, 2022

sonalgoyal May 19, 2022

navinrathore May 23, 2022

sonalgoyal May 19, 2022

navinrathore May 23, 2022

sonalgoyal May 19, 2022

navinrathore May 23, 2022

navinrathore commented May 23, 2022

navinrathore commented May 23, 2022

sonalgoyal May 23, 2022

navinrathore May 23, 2022

blockSize - a new config paramter for max size of the block #272

blockSize - a new config paramter for max size of the block #272

Conversation

navinrathore commented May 18, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

navinrathore commented May 23, 2022

navinrathore commented May 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment