Skip to content

vyeluri5/HPC-Data-Processing

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

HPC-Data-Processing

Repo to Initialise the MPI commands in C++ and process the large text files, which are of size 5GB or more.

Project context

The context of this whole code is to proccess a large textfile, generally of size 3GB or more and count the no of occurrence of a user give key word.

The main function takes two arguments textfilename.txt and key word to search through textfile and count no of occurrences. The for loop present in the code, to process large textfile (5GB in this case) by breaking it into smaller files (1GB min) and search through by adding all the words into a Vector words.

Once the message has been sent to slave nodes, each node will process the textfile defined by the user commands. In this case access the vector at a given index and count the occurrences of the key.

Finally, send all the caluculated information to Master node and write to a file.

About

Repo to Initialise the MPI commands in C++ and process the large text files which are of size 5GB or more.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors