Skip to content

@RafalGoslawski RafalGoslawski released this Nov 8, 2016

  • Fixed compatibility with yothalot library, splitsize can now longer be set (it now always is 10mb)
  • Refactored tempqueue to make it possible to add other feedback channels in the future (which we need for tcp based feedback)
  • The pool now works with the more abstract concept of handlers instead of rabbitmq connections, to allow the future additional of alternative feedback channels
  • Added first setup for the listener class, that allows feedback to be sent over tcp/ip instead of rabbitmq
  • When the tcp parameter is provided, it is now possible to wait for an answer via tcp on a random port
  • Removed recompress feature from the output::flush() method because it is also removed from the underlying c++ library
Assets 2

@RafalGoslawski RafalGoslawski released this Nov 1, 2016 · 7 commits to master since this release

  • prevent exceptions from bubbling up in case a directory does not exist
  • fixed extension to be compatible with updated yothalot library (but added a dependency on unavailable private copernica libraries)
  • updated default config file
  • allow datasize shortcuts (like 1MB) in ini file
  • the yothalot address can now be supplied in amqp:// format in the config file and as parameter to the job instead of supplying all parts individually
  • fixed compile errors
  • fixed wordcount example
  • work in progress on nosql support for the yothalot php extension
  • fixed a couple of todos for directory class
  • work in progress on using nosql for file storage
  • removed weird characters
  • fixed allocation issue
  • we now pass the nosql connection to the writer task, and the job::directory() method now always returns an directory that exists
  • the yothalot.maxcache ini setting is now parsed as human-readable data size (like "1MB")
  • fixed running finalization algorithm in client
  • the input files for writer tasks that are executed by the client are now automatically removed
  • no longer use readdir_r because it is going to be deprecated
  • the nodes now use the same cache settings that were used on the client that started the mapreduce process
  • renamed core class to rabbit class, added cache object to the connection class
  • fixed serialization for connection object
  • the cache settings that are passed to the yothalot connection object are now passed to the actual jobs, they no longer fetch settings from ini file
  • the local running finalizer now uses the cache settings from the connection object
  • cache object no longer holds a target object, because we do not need in in all places
  • the rabbit data is no longer serialized when a job is serialized
  • removed couple of todos
  • cache settings are now passed to the yothalot server
  • better error message
  • internal cancelled property in the tempqueue should be set earlier to prevent double cancels
  • for local finalizing, we need to change to the appropriate output dir
  • removed the TupleHelper class, and refactored it to seperate classes
  • added the RecordReduce class that allows to process records from the logfile directly
Assets 2

@RafalGoslawski RafalGoslawski released this Oct 3, 2016 · 39 commits to master since this release

  • First implementation of the Yothalot\Pool class, that allows to wait for the result of multiple jobs at the same time
  • Tempdir can now be set using the yothalot.temp-directory ini file setting (default = /tmp)
  • Added Descriptors class that encapsulates all descriptors used by a connection
  • Split loop and descriptors, to allow future support of multiple amqp connections
  • Allow callbacks to be passed to the event loop
  • json::object is internally already implemented as a sort of shared-ptr, no need to wrap it into another shared-ptr
  • Refactored stuff to make it easier to run multiple jobs in parallel, changed the return value for job::start()
  • Added method to fetch a job from a queue without waiting, added method to get pool size
  • Allow running the finalizer in the client
  • Tempqueue channel is now closed after result has been read, to fix the connection::flush() method that does not expect channels to survive
  • When a job is serialized/unserialized, it still is possible to run the finalizer locally
  • The _cancelled member was not initialized on construction
Assets 2

@RafalGoslawski RafalGoslawski released this Aug 1, 2016 · 55 commits to master since this release

  • Fix issue with getting the correct algorithm for serialized objects
  • Fix unserialize of job
Assets 2

@RafalGoslawski RafalGoslawski released this Jun 14, 2016 · 57 commits to master since this release

  • Added support for limiting record count in mapper tasks
  • Add option to set the split size for mapper input files and check whether the the maximum number of bytes is a multiple of this split size
  • Reworked error objects to extend from the regular result objects
  • Pass exception from missing mount point on to PHP userspace
  • Added Input\seek() method
  • Updated extension to new PHP-CPP api
  • Fixed the makefile for newer PHP versions
  • Removed dependency on React CPP
Assets 2

@RafalGoslawski RafalGoslawski released this Apr 4, 2016 · 63 commits to master since this release

  • Fix C++ exceptions that bubbled up to PHP space if a Yothalot\Output object was not correctly constructed
  • Use the pid of the process as a prefix to avoid getting the same 'unique' id, resulting in failure of directory creation
  • Update extension to be compatible with updated yothalot library (input object no longer supports start and size parameter, but a strict parameter)
  • Add dependencies to makefile
  • Add the recompress option to the Yothalot\Output::flush method
Assets 2

@RafalGoslawski RafalGoslawski released this Mar 22, 2016 · 70 commits to master since this release

  • We no longer have a version 1 and 2 of mapreduce, there's a single MapTask now.
  • Added backwards compatilibity for the kvmapper
  • Added a Yothalot\Result interface for us to implement
  • A Yothalot\Input doesn't have to be a log file.. just a yothalot file
  • Fix issue with add and race jobs
Assets 2

@RafalGoslawski RafalGoslawski released this Mar 16, 2016 · 76 commits to master since this release

  • fix issue with flushing an output object
  • remove unused task property
  • add a member to input to check if the input file is valid
  • add next method to input class
  • make version 1 default if version is not set
Assets 2

@RafalGoslawski RafalGoslawski released this Mar 1, 2016 · 83 commits to master since this release

  • Take into account that we could have an error object in our result json
Assets 2

@RafalGoslawski RafalGoslawski released this Mar 1, 2016 · 85 commits to master since this release

  • Removed calls to Php::Namespace class (it appears to be broken)
  • Implemented an ErrorResult object
  • Don't emit empty warnings because that doesn't make much sense
Assets 2
You can’t perform that action at this time.