Support for Header Substring

[No1e]

Hello,

Is there a way to configure CSVReader to take only portion of CSV header?

I am using generic way for parsing, like this:

CsvReader csvReader = new CsvReader(reader, csvConfig)
foreach (var assetRecord in csvReader.GetRecords<dynamic>())

In the CSV file, in the header I have something like this Name1(2,3) | Name2(4,2)...

So, instead taking Name1(1,3), I would like to take part of column name up to brackets Name1.

Is this possible?

Regards,
Novak

[JoshClose]

void Main()
{
	using (var stream = new MemoryStream())
	using (var writer = new StreamWriter(stream))
	using (var reader = new StreamReader(stream))
	using (var csv = new CsvReader(reader))
	{
		writer.WriteLine("Id(1),Name(2)");
		writer.WriteLine("1,one");
		writer.WriteLine("2,two");
		writer.Flush();
		stream.Position = 0;
		
		csv.Configuration.PrepareHeaderForMatch = header => header.Substring(0, header.IndexOf("("));
		csv.GetRecords<dynamic>().ToList().Dump();
	}
}

[No1e]

Cool thanks.

Is there any way to improve performance for this? I have csv file that contains close to 100K records. And on mobile device it takes 23 for this:
csv.GetRecords().ToList()

[JoshClose]

Don't use dynamic. There is an issue in to speed it up, but right now it's pretty slow. Use a normal class or just calling GetField<int>("Field") is really fast.

[No1e]

Ok. But the problem is that I have to handle that in generic way, so I can not use normal class not calling GetField<int>("Field") or similar.

Do you have on the road map performance improvements for this use case?

[JoshClose]

You should be able to handle reading in a generic way easier using GetField than using dynamic. Once you have the dynamic object, how are you getting the data from the properties?

[No1e]

The thing is that because of the requirements, the whole model is completely generic, name of the fields are not known, can be anything, whatever user puts in the CSV file. That is the reason why I am using dynamic. Later on I access each property from dynamic object, which I also do not like very much, but currently I do not have other solution.

[JoshClose]

Can you give an example of how you're getting the data out of the dynamic object? Maybe I can translate that into using CsvHelper directly to do it instead, to speed it up.

[No1e]

Sure I can. Here is how it looks:

foreach (var record in csvReader.GetRecords<dynamic>()) { foreach (var recordProperty in record) {

Please note that calling Get records takes approx. 23 seconds.

[JoshClose]

If your'e doing GetRecords (plural), that is pulling the entire file into memory.

Try doing it like this. It should be a lot faster.

void Main()
{
	using (var stream = new MemoryStream())
	using (var writer = new StreamWriter(stream))
	using (var reader = new StreamReader(stream))
	using (var csv = new CsvReader(reader))
	{
		writer.WriteLine("Id(1),Name(2)");
		writer.WriteLine("1,one");
		writer.WriteLine("2,two");
		writer.Flush();
		stream.Position = 0;
		
		csv.Configuration.PrepareHeaderForMatch = header => header.Substring(0, header.IndexOf("("));
		csv.Read();
		csv.ReadHeader();
		while (csv.Read())
		{
			for (var i = 0; i < csv.Context.HeaderRecord.Length; i++)
			{
				var field = csv.GetField(i);
			}
		}
	}
}

public class Test
{
	public int Id { get; set; }
	public string Name { get; set; }
}

public class TestMap : ClassMap<Test>
{
	public TestMap()
	{
		Map(m => m.Id);
		Map(m => m.Name);
	}
}

[augustoproiete]

If you're doing GetRecords (plural), that is pulling the entire file into memory.

@JoshClose I'm probably missing something, but wouldn't the example code above, calling Read manually yourself, produce the same memory usage than calling GetRecords, given that GetRecords streams the records to the caller?

And with that said, in both cases you don't actually have the entire file into memory at any one point, as previous records get GC'ed...

[JoshClose]

Sorry, you are correct. It yields records and there is no ToList or anything.

The speed improvements come from not using dynamic.

[No1e]

Hi Josh.

Thanks. I give it a try and get back to you with results.