campt return contains processing logs

[michaelaye]

I'm trying to receive a clean pvl object from campt but the output is polluted by processing logs.

Here's my campt call, for reference:

pvlstr = campt(from_=fname, format='pvl', type='image', allowoutside=False,
               sample=x, line=y)

and here's the content of pvlstr, after decoding:

['campt: Working',
 '0% Processed',
 'Group = GroundPoint',
 '  Filename                   = /Volumes/Data/p4terrains/G11_022679_0981_XN_81-',
 '                               S218W/G11_022679_0981_XN_81S218W.cal.cub',
 '  Sample                     = 2500.0',
 '  Line                       = 9728.0',
 '  PixelValue                 = 0.083468907',
 '  RightAscension             = 332.06993073767',
 '  Declination                = 45.467397759472',
 '  PlanetocentricLatitude     = -81.900874169456',
 '  PlanetographicLatitude     = -81.994697096444',
 '  PositiveEast360Longitude   = 141.73099478747',
 '  PositiveEast180Longitude   = 141.73099478747',
 '  PositiveWest360Longitude   = 218.26900521253',
 '  PositiveWest180Longitude   = -141.73099478747',
 '  BodyFixedCoordinate        = (-374.02263821935, 295.05689330782,',
 '                                -3347.6867855901) <km>',
 '  LocalRadius                = 3381413.6568311 <meters>',
 '  SampleResolution           = 5.0364890282903 <meters/pixel>',
 '  LineResolution             = 5.0364890282903 <meters/pixel>',
 '',
 '  # Spacecraft Information',
 '  SpacecraftPosition         = (-349.05484937137, 341.30534280218,',
 '                                -3596.1186900101) <km>',
 '  SpacecraftAzimuth          = 179.68727531586',
 '  SlantDistance              = 253.93054305131 <km>',
 '  TargetCenterDistance       = 3629.104332688 <km>',
 '  SubSpacecraftLatitude      = -82.269103182543',
 '  SubSpacecraftLongitude     = 135.64313582454',
 '  SpacecraftAltitude         = 248.36732051168 <km>',
 '  OffNadirAngle              = 12.278580772889',
 '  SubSpacecraftGroundAzimuth = 243.29165393851',
 '',
 '  # Sun Information',
 '  SunPosition                = (29208379.379094, 196548117.44241,',
 '                                -77404100.772068) <km>',
 '  SubSolarAzimuth            = 130.66848906188',
 '  SolarDistance              = 1.4254805528648 <AU>',
 '  SubSolarLatitude           = -21.282891597047',
 '  SubSolarLongitude          = 81.547320362307',
 '  SubSolarGroundAzimuth      = 296.7532998578',
 '',
 '  # Illumination and Other',
 '  Phase                      = 57.55877071508',
 '  Incidence                  = 64.873786916827',
 '  Emission                   = 13.193650088776',
 '  NorthAzimuth               = 59.299690450996',
 '',
 '  # Time',
 '  EphemerisTime              = 359955489.5936 <seconds>',
 '  UTC                        = 2011-05-29T15:37:03.408633',
 '  LocalSolarTime             = 16.012244961677 <hour>',
 '  SolarLongitude             = 301.4927849252',
 'End_Group',
 '5% Processed',
 '10% Processed',
 '15% Processed',
 '20% Processed',
 '25% Processed',
 '30% Processed',
 '35% Processed',
 '40% Processed',
 '45% Processed',
 '50% Processed',
 '55% Processed',
 '60% Processed',
 '65% Processed',
 '70% Processed',
 '75% Processed',
 '80% Processed',
 '85% Processed',
 '90% Processed',
 '95% Processed',
 '100% Processed']

So, currently, to parse this with your pvl module I need to do hackery like this to get it to work:

pvl.loads('\n'.join(pvlstr.decode().splitlines()[2:-20]))

I'm guessing this is maybe mixing STDOUT and STDERR somehow? Or maybe not, but is there a way to get only the clean PVL object returned?

[wtolson]

This is a tricky one. It's definitely writing both the progress and the point label to stdout. One solution would be to create a utility to parse this out. A second option you can test out is it appears you can create a file "$HOME/.Isis/IsisPreferences" and add the following to suppress the progress output:

Group=UserInterface
  ProgressBar        = Off
EndGroup

Ideally isis would be outputting the progress to STDERR but that's probably not a realistic solution.

[michaelaye]

That did it, thanks!
A thought on the output: it looks like it's using both STDOUT and STDERR because of the odd timing: First, one get's part of the progress bar, then the results, then the rest of the progress bar. This looks like the PVL is pushed a different stream than the progress bar, and possibly pysis just would need to differentiate between those?

[wtolson]

Yep, I went through the source of campt in isis (version 3.4.10.6373) and that's just the weird way it prints it out, all of it going to STDOUT. In pysis only stdout is returned (stderr attached to an exception if the command returns a non zero status code).

[michaelaye]

I will report that as another error to the ISIS team, as I’m already reporting the incorrect treatment of the allowoutside option.

On Apr 1, 2016, at 11:30, Trevor notifications@github.com wrote:

Yep, I went threw the source of campt in isis (version 3.4.10.6373) and that's just the weird way it prints it out, all of it going to STDOUT. In pysis only stdout is returned (stderr attached to an exception if the command returns a non zero status code).

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub #16 (comment)

[michaelaye]

Hm, if everything goes to STDOUT, how can the result be in the stream earlier than the pvl? Is that weird Python buffering then?

[wtolson]

Here's the relevant code from isis/src/base/apps/campt/campt.cpp:

// Write our point coordinates to std out in pvl format, or to a pvl or some type of flat file
void writePoints(const UserInterface &ui, QList<PvlGroup*> camPoints) {
  // Progress should increment for each point we process
  Progress prog;
  prog.SetMaximumSteps(camPoints.size());

  // ... removed for length

  for (int p = 0; p < camPoints.size(); p++) {
    bool fileExists = FileName(outFile).fileExists();

    prog.CheckStatus();
    point = camPoints[p];

    // ... removed for length

    // we still want to output the results
    Application::Log((*point));
    delete point;
    point = NULL;
  }
  prog.CheckStatus();
}

Basically what's happening is it initializes the progress bar which outputs %0 and then prints the 1 point. The weird part is the progress bar then prints out the rest of the output all at once even though everything has already completed. From isis/src/base/objs/Progress/Progress.cpp in void Progress::CheckStatus():

// See if the percent processed needs to be updated
while(100.0 * p_currentStep / p_maximumSteps >= p_currentPercent) {
  if(Isis::iApp != NULL) {
    if (p_autoDisplay) {
      Isis::iApp->UpdateProgress(p_currentPercent, p_printPercent);
    }
  }
  else {
    if(p_printPercent && p_autoDisplay) {
      if(p_currentPercent < 100) {
        cout << p_currentPercent << "% Processed\r" << flush;
      }
      else {
        cout << p_currentPercent << "% Processed" << endl;
      }
    }
  }
  p_currentPercent += p_percentIncrement;
}