Look into making FluoConfiguration Serializable #813

Closed
keith-turner opened this Issue Dec 16, 2016 · 1 comment

Comments

Projects
None yet
2 participants
@keith-turner
Contributor

keith-turner commented Dec 16, 2016

Was looking into writing a Spark code using Fluo. I was expecting FluoConfiguration to be serializable and it was not.

@keith-turner

This comment has been minimized.

Show comment
Hide comment
@keith-turner

keith-turner Dec 19, 2016

Contributor

In the next release of Fluo, FluoConfiguration will be serializable. There are many ways to work around this issue before the next release. Below is an example of one way.

void doFluoStuff(FluoConfiguration fconf, JavaRDD<?> someRdd){
  //this map is serializable
  Map<String, String> confMap = fconf.toMap();

  someRdd.foreachPartition(iter->{
    FluoConfiguration fconf = new FluoConfiguration(confMap);
    try(FluoClient client = FluoFactory.newClient(fconf)){
      //do fluo stuff
    }
  }
}
Contributor

keith-turner commented Dec 19, 2016

In the next release of Fluo, FluoConfiguration will be serializable. There are many ways to work around this issue before the next release. Below is an example of one way.

void doFluoStuff(FluoConfiguration fconf, JavaRDD<?> someRdd){
  //this map is serializable
  Map<String, String> confMap = fconf.toMap();

  someRdd.foreachPartition(iter->{
    FluoConfiguration fconf = new FluoConfiguration(confMap);
    try(FluoClient client = FluoFactory.newClient(fconf)){
      //do fluo stuff
    }
  }
}

@asfgit asfgit closed this in bf13386 Dec 20, 2016

@mikewalch mikewalch added this to the 1.1.0 milestone May 19, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment